Towards Unified Vision Language Models for Forest Ecological Analysis in Earth Observation

TL;DR

This work addresses the need for benchmarks that unify descriptive reasoning and quantitative regression in Earth Observation using vision-language models. It introduces REO-Instruct, a large-scale multimodal benchmark pairing co-registered Sentinel-2 multispectral data and ALOS-2 SAR with domain-rich text to support land-cover classification, ecological patch counting, VQA on human activity, and above-ground biomass (AGB) regression, with 1.6M training, ~20k validation, and ~36k test samples. The authors propose a cognitively interpretable logical-chain framework and provide data collection, annotation, and prompt-design pipelines, including automatic quality checks and expert review. Experimental results show current VLMs excel at content understanding but struggle with numeric reasoning, highlighting the need for regression-aware training and multimodal integration to enable reliable forest ecological inference. Overall, REO-Instruct establishes a standardized foundation for advancing scientifically grounded, generation-plus-regression vision-language systems in forest ecology and related geoscience domains.

Abstract

Recent progress in vision language models (VLMs) has enabled remarkable perception and reasoning capabilities, yet their potential for scientific regression in Earth Observation (EO) remains largely unexplored. Existing EO datasets mainly emphasize semantic understanding tasks such as captioning or classification, lacking benchmarks that align multimodal perception with measurable biophysical variables. To fill this gap, we present REO-Instruct, the first unified benchmark designed for both descriptive and regression tasks in EO. REO-Instruct establishes a cognitively interpretable logic chain in forest ecological scenario (human activity,land-cover classification, ecological patch counting, above-ground biomass (AGB) regression), bridging qualitative understanding and quantitative prediction. The dataset integrates co-registered Sentinel-2 and ALOS-2 imagery with structured textual annotations generated and validated through a hybrid human AI pipeline. Comprehensive evaluation protocols and baseline results across generic VLMs reveal that current models struggle with numeric reasoning, highlighting an essential challenge for scientific VLMs. REO-Instruct offers a standardized foundation for developing and assessing next-generation geospatial models capable of both description and scientific inference. The project page are publicly available at \href{https://github.com/zhu-xlab/REO-Instruct}{REO-Instruct}.

Figures (5)

• Figure 1: (a). Hierarchical structure of VLM capabilities: From basic perception tasks to higher-order reasoning tasks; (b). Advantages of VLM for EO regression tasks: By integrating scientific domain knowledge with EO image data, VLMs overcome the information bottleneck of traditional image-only regression models, enabling deeper insights and improved scientific reasoning; (c). Interplay between regression and generation tasks: Using AGB estimation as an example, the intrinsic link between regression and generation targets allows collaborative processing in a unified framework, enhancing prediction accuracy and reliability.
• Figure 2: (a). RGB modality examples and word cloud of REO-Instruct benchmark; (b). Screenshots of some image-texts annotation pairs in REO-Instruct benchmark.
• Figure 3: Land cover distribution based on the number of samples in each category.
• Figure 4: Distribution of Above-Ground Biomass (AGB) values in REO-Instruct. The histogram shows the frequency of AGB values.
• Figure 5: Test guiding prompts for compared methods.