Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OmniGeo: Towards a Multimodal Large Language Models for Geospatial Artificial Intelligence

Long Yuan, Fengran Mo, Kaiyu Huang, Wenjie Wang, Wangyuxuan Zhai, Xiaoyu Zhu, You Li, Jinan Xu, Jian-Yun Nie

TL;DR

OmniGeo targets the multimodal GeoAI bottleneck by unifying text, imagery, and geospatial data into a single large language model tailored for geospatial tasks. Through multimodal instruction data construction and joint fine-tuning (including LoRA and full-parameter methods) across five GeoAI tasks, OmniGeo demonstrates robust cross-domain reasoning and competitive or superior performance relative to task-specific models and GPT-4o on several benchmarks. The work provides 12 multimodal instruction datasets, shows substantial gains from multimodal and multi-task training, and proposes a practical path toward deploying a single, versatile GeoAI model. This approach has potential implications for scalable GeoAI applications across health geography, urban planning, remote sensing, and geospatial semantics, enabling richer, grounded geospatial reasoning in real-world tasks.

Abstract

The rapid advancement of multimodal large language models (LLMs) has opened new frontiers in artificial intelligence, enabling the integration of diverse large-scale data types such as text, images, and spatial information. In this paper, we explore the potential of multimodal LLMs (MLLM) for geospatial artificial intelligence (GeoAI), a field that leverages spatial data to address challenges in domains including Geospatial Semantics, Health Geography, Urban Geography, Urban Perception, and Remote Sensing. We propose a MLLM (OmniGeo) tailored to geospatial applications, capable of processing and analyzing heterogeneous data sources, including satellite imagery, geospatial metadata, and textual descriptions. By combining the strengths of natural language understanding and spatial reasoning, our model enhances the ability of instruction following and the accuracy of GeoAI systems. Results demonstrate that our model outperforms task-specific models and existing LLMs on diverse geospatial tasks, effectively addressing the multimodality nature while achieving competitive results on the zero-shot geospatial tasks. Our code will be released after publication.

OmniGeo: Towards a Multimodal Large Language Models for Geospatial Artificial Intelligence

TL;DR

OmniGeo targets the multimodal GeoAI bottleneck by unifying text, imagery, and geospatial data into a single large language model tailored for geospatial tasks. Through multimodal instruction data construction and joint fine-tuning (including LoRA and full-parameter methods) across five GeoAI tasks, OmniGeo demonstrates robust cross-domain reasoning and competitive or superior performance relative to task-specific models and GPT-4o on several benchmarks. The work provides 12 multimodal instruction datasets, shows substantial gains from multimodal and multi-task training, and proposes a practical path toward deploying a single, versatile GeoAI model. This approach has potential implications for scalable GeoAI applications across health geography, urban planning, remote sensing, and geospatial semantics, enabling richer, grounded geospatial reasoning in real-world tasks.

Abstract

The rapid advancement of multimodal large language models (LLMs) has opened new frontiers in artificial intelligence, enabling the integration of diverse large-scale data types such as text, images, and spatial information. In this paper, we explore the potential of multimodal LLMs (MLLM) for geospatial artificial intelligence (GeoAI), a field that leverages spatial data to address challenges in domains including Geospatial Semantics, Health Geography, Urban Geography, Urban Perception, and Remote Sensing. We propose a MLLM (OmniGeo) tailored to geospatial applications, capable of processing and analyzing heterogeneous data sources, including satellite imagery, geospatial metadata, and textual descriptions. By combining the strengths of natural language understanding and spatial reasoning, our model enhances the ability of instruction following and the accuracy of GeoAI systems. Results demonstrate that our model outperforms task-specific models and existing LLMs on diverse geospatial tasks, effectively addressing the multimodality nature while achieving competitive results on the zero-shot geospatial tasks. Our code will be released after publication.

Paper Structure

This paper contains 23 sections, 8 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Task Definition
  5. Exploration of LLMs for GeoAI
  6. Methodology
  7. Multimodal Instruction Data Construction
  8. Image-Text Instruction Fine-Tuning
  9. Training Objective
  10. Experiments
  11. Datasets
  12. Evaluation Metrics
  13. Baseline Methods
  14. Implementation Details
  15. Results
  16. ...and 8 more sections

Figures (6)

  • Figure 1: The Illustration of all the tasks covered by OmniGeo through an engaging dialogue.
  • Figure 2: A detailed illustration of the image-text instruction data and geospatial tasks covered by OmniGeo.
  • Figure 3: The training framework overview of the OmniGeo, which integrates five GeoAI tasks with various modalities.
  • Figure 4: Prediction error plot for each baseline and OmniGeo (LLaVA) on the dementia deaths time series forecasting task. The color of each US region indicates the percentage error of each model in predicting that state $PE = (Prediction - True)/True$.
  • Figure 5: Confusion matrix comparison of Place2Vec, HGI, Blip2, GPT-4o and OmniGeo (LLaVA) models on the Shenzhen urban region function classification dataset
  • ...and 1 more figures