VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing

TL;DR

VLRS-Bench tackles a key gap in remote sensing by introducing a cognition-driven benchmark for multimodal reasoning, organized into Cognition, Decision, and Prediction with 14 L-3 tasks and 2,000 QA pairs generated via RS priors (DSM, NIR) and expert masks. The authors present a three-tier pipeline—data assembly, instruction synthesis, and rigorous verification (automated checks, cross-model validation, and expert review)—to ensure geospatial realism and reasoning depth. Across experiments, general MLLMs show bottlenecks in geospatial reasoning, while RS-specialized models fare better but still struggle on planning and long-horizon prediction, underscoring the need for RS-aware architectures. By exposing nuanced strengths and weaknesses through L-1 to L-3 analyses and a robust verification framework, VLRS-Bench provides a rigorous platform to drive principled improvements in remote sensing multimodal reasoning.

Abstract

Recent advancements in Multimodal Large Language Models (MLLMs) have enabled complex reasoning. However, existing remote sensing (RS) benchmarks remain heavily biased toward perception tasks, such as object recognition and scene classification. This limitation hinders the development of MLLMs for cognitively demanding RS applications. To address this, , we propose a Vision Language ReaSoning Benchmark (VLRS-Bench), which is the first benchmark exclusively dedicated to complex RS reasoning. Structured across the three core dimensions of Cognition, Decision, and Prediction, VLRS-Bench comprises 2,000 question-answer pairs with an average length of 71 words, spanning 14 tasks and up to eight temporal phases. VLRS-Bench is constructed via a specialized pipeline that integrates RS-specific priors and expert knowledge to ensure geospatial realism and reasoning complexity. Experimental results reveal significant bottlenecks in existing state-of-the-art MLLMs, providing critical insights for advancing multimodal reasoning within the remote sensing community.

Figures (25)

• Figure 1: Pipeline for constructing VLRS-Bench. The process integrates the target RGB image with multi-source remote sensing priors (e.g., DSM and expert masks) to form a structured multimodal instruction, which guides a GPT-5-chat to produce reasoning tasks across cognitive dimensions. Each generated item is then verified through a three-stage protocol, including automated filtering, multi-MLLM cross-validation, and human expert review.
• Figure 1: An illustration of the Mask_Info stage using the GID-15 dataset's mask palette as an example. This process acts as a “ semantic bridge,” transforming a raw segmentation mask into a structured, semantically rich instruction. For each class in the GID-15 palette, the pipeline maps the pixel color to a standardized hexadecimal code and appends a functional description, bridging the semantic gap and enabling expert-level reasoning about land use.
• Figure 2: Avg. Score of various MLLMs across four QA-types. The distinct color coding (e.g. Qwen2.5-VL-32B in Blue, GPT-4o-mini in Yellow) highlights a critical phenomenon: a sharp performance drop from Single-Choice to Multi-Choice and Fill in Blank tasks. This trend, consistent across model sizes, validates the high reasoning ceiling of VLRS-Bench.
• Figure 2: An example of the Dataset Info component for the LoveDA dataset. This text provides the MLLMs with high-level contextual information, including geographic origin, ground resolution, and typical scene types. This meta-level knowledge is crucial for enabling more plausible and context-aware reasoning.
• Figure 3: The template for the Format-Constrained Prompt. This template enforces a strict JSON schema and a set of validation rules, ensuring that all generated evaluation items are structurally consistent, machine-readable, and adhere to predefined quality standards.