[📜 Paper] • [🤗 HF Dataset] • [🐱 GitHub Code]
- 2024/12/5: 🔥GitHub Code repo (GeoMap-Agent) released.
- 2024/12/5: 🔥HuggingFace Dataset repo (GeoMap-Bench) released.
- 2024/12/5: 🔥Arxiv paper (PEACE) released.
Geologic map, as a fundamental diagram in geology science, provides critical insights into the structure and composition of Earth's subsurface and surface. These maps are indispensable in various fields, including disaster detection, resource exploration, and civil engineering.
We present GeoMap-Bench, a new benchmark consisting of 124 geologic maps and 3,864 multimodal multiple-choice questions with diverse annotations. The distribution of questions in the GeoMap-Bench. It consists of 25 task types that measure critical geological map interpretation abilities across five aspects: grounding, extracting, referring, reasoning, and analyzing.
We introduce GeoMap-Agent, the inaugural agent designed for geologic map understanding, which features three modules: Hierarchical Information Extraction(HIE), Domain Knowledge Injection (DKI), and Prompt-enhanced Question Answering (PEQA). Inspired by the interdisciplinary collaboration among human scientists, an AI expert group acts as consultants, utilizing a diverse tool pool to comprehensively analyze questions.
Through comprehensive experiments, GeoMap-Agent achieves an overall score of 0.811 on GeoMap-Bench, significantly outperforming 0.369 of GPT-4o.
| Method | Extracting | Grounding | Referring | Reasoning | Analyzing | Overall |
|---|---|---|---|---|---|---|
| Random | 0 | 0 | 0.250 | 0.250 | 0 | 0.100 |
| GPT-4o | 0.219 | 0.128 | 0.378 | 0.507 | 0.612 | 0.369 |
| GeoMap-Agent | 0.832 | 0.920 | 0.886 | 0.588 | 0.831 | 0.811 |
We visualize the components of a typical geologic map, highlighting the complex nature of cartographic generalization. Additionally, we showcase sample questions from GeoMap-Bench and demonstrate how our GeoMap-Agent integrates various sources of contextual information to address them.
Installation
- Step1: Clone GeoMap-Agent code repository
git clone https://github.com/microsoft/PEACE.git
cd PEACE
- Step2: Clone GeoMap-Bench dataset repository
git lfs install
git clone https://huggingface.co/datasets/microsoft/PEACE data
- Step3: Download layout detection models
wget -O models.zip https://xqocha.bn.files.1drv.com/y4madviPzB-SHaZ6mqGYhJskMw7q08hW368t0bIQRR3hJbIkv3pe8DdE5As3ioUkUg_2wmiygGve-FA7lKX6bjqyPM8inYO104zsdQViY-d6NDKgvgWXiqTX0n-PmXteh0xTwdO2oUfEKycLSzPYhB1Z1SkjUiRDfJksMGb-5iMfb7vu3mp4V-g62rMhzYweiv6rMQ5zkOMqh-Y1JnRhkzaVTuBgJeemG4BAdMVj73a4QI
unzip models.zip -d dependencies
- Step4: Install dependencies
pip install -r requirements.txt
- Step5: Configure LLMs API endpoint and key in utils/api.py
Evaluation
python eval.py --copilot_mode HIE,DKI,PEQA --dataset_source usgs
Due to the inherent limitations of large language models, issues such as hallucinations may occur.
TBD
This repository is licensed under the MIT License. The use of Ultralytics library is subject to the AGPL-3.0 License.