Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretation

Zhenshi Li, Dilxat Muhtar, Feng Gu, Xueliang Zhang, Pengfeng Xiao, Guangjun He, Xiaoxiang Zhu

TL;DR

LHRS-Bot-Nova is introduced, an MLLM specialized in understanding remote sensing images, designed to expertly perform a wide range of RS understanding tasks aligned with human instructions, and an instruction dataset specifically designed to improve spatial recognition abilities is introduced.

Abstract

Automatically and rapidly understanding Earth's surface is fundamental to our grasp of the living environment and informed decision-making. This underscores the need for a unified system with comprehensive capabilities in analyzing Earth's surface to address a wide range of human needs. The emergence of multimodal large language models (MLLMs) has great potential in boosting the efficiency and convenience of intelligent Earth observation. These models can engage in human-like conversations, serve as unified platforms for understanding images, follow diverse instructions, and provide insightful feedbacks. In this study, we introduce LHRS-Bot-Nova, an MLLM specialized in understanding remote sensing (RS) images, designed to expertly perform a wide range of RS understanding tasks aligned with human instructions. LHRS-Bot-Nova features an enhanced vision encoder and a novel bridge layer, enabling efficient visual compression and better language-vision alignment. To further enhance RS-oriented vision-language alignment, we propose a large-scale RS image-caption dataset, generated through feature-guided image recaptioning. Additionally, we introduce an instruction dataset specifically designed to improve spatial recognition abilities. Extensive experiments demonstrate superior performance of LHRS-Bot-Nova across various RS image understanding tasks. We also evaluate different MLLM performances in complex RS perception and instruction following using a complicated multi-choice question evaluation benchmark, providing a reliable guide for future model selection and improvement. Data, code, and models will be available at https://github.com/NJU-LHRS/LHRS-Bot.

LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretation

TL;DR

LHRS-Bot-Nova is introduced, an MLLM specialized in understanding remote sensing images, designed to expertly perform a wide range of RS understanding tasks aligned with human instructions, and an instruction dataset specifically designed to improve spatial recognition abilities is introduced.

Abstract

Automatically and rapidly understanding Earth's surface is fundamental to our grasp of the living environment and informed decision-making. This underscores the need for a unified system with comprehensive capabilities in analyzing Earth's surface to address a wide range of human needs. The emergence of multimodal large language models (MLLMs) has great potential in boosting the efficiency and convenience of intelligent Earth observation. These models can engage in human-like conversations, serve as unified platforms for understanding images, follow diverse instructions, and provide insightful feedbacks. In this study, we introduce LHRS-Bot-Nova, an MLLM specialized in understanding remote sensing (RS) images, designed to expertly perform a wide range of RS understanding tasks aligned with human instructions. LHRS-Bot-Nova features an enhanced vision encoder and a novel bridge layer, enabling efficient visual compression and better language-vision alignment. To further enhance RS-oriented vision-language alignment, we propose a large-scale RS image-caption dataset, generated through feature-guided image recaptioning. Additionally, we introduce an instruction dataset specifically designed to improve spatial recognition abilities. Extensive experiments demonstrate superior performance of LHRS-Bot-Nova across various RS image understanding tasks. We also evaluate different MLLM performances in complex RS perception and instruction following using a complicated multi-choice question evaluation benchmark, providing a reliable guide for future model selection and improvement. Data, code, and models will be available at https://github.com/NJU-LHRS/LHRS-Bot.

Paper Structure

This paper contains 17 sections, 4 equations, 7 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. MLLM development in the RS community
  4. Large-scale RS vision-language dataset
  5. Dataset
  6. LHRS-Align-Recap: feature-guided image recaptioning for multimodal alignment
  7. Instruction tuning datasets for training LHRS-Bot-Nova
  8. Methodology
  9. Model architecture
  10. Training strategy
  11. Experiments
  12. Experimental setup
  13. Quantitative results on RS image understanding
  14. Evaluation on LHRS-Bench
  15. Multimodal dialogue in RS understanding
  16. ...and 2 more sections

Figures (7)

  • Figure 1: Four examples from LHRS-Align and LHRS-Align-Recap.
  • Figure 2: Caption length (left) and image-caption LongCLIP score (right) distributions for the LHRS-Align and the proposed LHRS-Align-Recap dataset.
  • Figure 3: Six examples of various instruction tasks from the instruction datasets used to train LHRS-Bot-Nova.
  • Figure 4: Architecture of LHRS-Bot-Nova. LHRS-Bot-Nova employs learnable queries in conjunction with an MoE vision perceiver to summarize multi-level visual representations, which are then concatenated with language token embeddings as input to the LLM.
  • Figure 5: Hyperparameter settings for each stage.
  • ...and 2 more figures