Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Tool-Augmented Agents in Remote Sensing Platforms

Simranjit Singh, Michael Fore, Dimitrios Stamoulis

TL;DR

GeoLLM-QA addresses the need for evaluating tool-augmented LLMs in remote sensing under realistic, user-grounded tasks rather than template-based QA. It introduces a 1,000-task benchmark executed on a live geospatial UI with a 117-tool toolkit and three RS datasets, using oracle detectors to isolate agent decisions from detector errors. The benchmark uses a three-step data-generation process (reference templates, LLM-guided perturbations, and ground-truth generation with retrieval augmentation) and a multi-metric evaluation suite including success rate, correctness, ROUGE-L, token cost, and recall. Key findings show GPT-4 Turbo with function-calling delivers strong performance, CoT and ReAct generally outperform Chameleon, ROUGE-L varies with prompting, and recall reveals gaps in detection; the work provides a foundation for developing stronger geospatial copilots and emphasizes open-sourcing and future directions such as replacing detectors and reducing human-in-the-loop overhead.

Abstract

Tool-augmented Large Language Models (LLMs) have shown impressive capabilities in remote sensing (RS) applications. However, existing benchmarks assume question-answering input templates over predefined image-text data pairs. These standalone instructions neglect the intricacies of realistic user-grounded tasks. Consider a geospatial analyst: they zoom in a map area, they draw a region over which to collect satellite imagery, and they succinctly ask "Detect all objects here". Where is `here`, if it is not explicitly hardcoded in the image-text template, but instead is implied by the system state, e.g., the live map positioning? To bridge this gap, we present GeoLLM-QA, a benchmark designed to capture long sequences of verbal, visual, and click-based actions on a real UI platform. Through in-depth evaluation of state-of-the-art LLMs over a diverse set of 1,000 tasks, we offer insights towards stronger agents for RS applications.

Evaluating Tool-Augmented Agents in Remote Sensing Platforms

TL;DR

GeoLLM-QA addresses the need for evaluating tool-augmented LLMs in remote sensing under realistic, user-grounded tasks rather than template-based QA. It introduces a 1,000-task benchmark executed on a live geospatial UI with a 117-tool toolkit and three RS datasets, using oracle detectors to isolate agent decisions from detector errors. The benchmark uses a three-step data-generation process (reference templates, LLM-guided perturbations, and ground-truth generation with retrieval augmentation) and a multi-metric evaluation suite including success rate, correctness, ROUGE-L, token cost, and recall. Key findings show GPT-4 Turbo with function-calling delivers strong performance, CoT and ReAct generally outperform Chameleon, ROUGE-L varies with prompting, and recall reveals gaps in detection; the work provides a foundation for developing stronger geospatial copilots and emphasizes open-sourcing and future directions such as replacing detectors and reducing human-in-the-loop overhead.

Abstract

Tool-augmented Large Language Models (LLMs) have shown impressive capabilities in remote sensing (RS) applications. However, existing benchmarks assume question-answering input templates over predefined image-text data pairs. These standalone instructions neglect the intricacies of realistic user-grounded tasks. Consider a geospatial analyst: they zoom in a map area, they draw a region over which to collect satellite imagery, and they succinctly ask "Detect all objects here". Where is `here`, if it is not explicitly hardcoded in the image-text template, but instead is implied by the system state, e.g., the live map positioning? To bridge this gap, we present GeoLLM-QA, a benchmark designed to capture long sequences of verbal, visual, and click-based actions on a real UI platform. Through in-depth evaluation of state-of-the-art LLMs over a diverse set of 1,000 tasks, we offer insights towards stronger agents for RS applications.
Paper Structure (4 sections, 3 figures, 1 table)

This paper contains 4 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. The GeoLLM-QA Framework
  3. Experiments
  4. Conclusion and Future Work

Figures (3)

  • Figure 1: Unlike prior work that assumes task-specific templates (e.g., "[detection]" keyword), GeoLLM-QA requires the agent to follow nuanced instructions and perform multi-step reasoning to accomplish user-defined objectives.
  • Figure 2: GeoLLM-QA challenges agents to solve complex RS tasks through multimodal reasoning and actions over long sequences of verbal, visual, and click-based actions on a real UI platform.
  • Figure 3: GPT-3.5 vs. GPT-4 error analysis for CoT prompting.