Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IC-EO: Interpretable Code-based assistant for Earth Observation

Lamia Lahouel, Laurynas Lopata, Simon Gruening, Gabriele Meoni, Gaetan Petit, Sylvain Lobry

TL;DR

This work presents IC--EO, a code-first conversational agent that converts natural-language Earth Observation queries into executable Python workflows via a modular API of EO tools. By coupling GPT--4o with a structured tool registry and sandboxed execution, IC--EO delivers auditable, reproducible pipelines for tasks such as land-cover mapping and post-wildfire damage assessment, outperforming general-purpose VLM baselines on these use-cases. The evaluation spans tool-level, agent-level, and task-level metrics, highlighting high executable-code rates and transparent reasoning, while identifying memory and toolchain constraints as the main sources of failure. Overall, the study demonstrates that structured, code-generated EO workflows can achieve greater interpretability and reliability than end-to-end multimodal models, with strong potential for extensibility as new tools and datasets are added.

Abstract

Despite recent advances in computer vision, Earth Observation (EO) analysis remains difficult to perform for the laymen, requiring expert knowledge and technical capabilities. Furthermore, many systems return black-box predictions that are difficult to audit or reproduce. Leveraging recent advances in tool LLMs, this study proposes a conversational, code-generating agent that transforms natural-language queries into executable, auditable Python workflows. The agent operates over a unified easily extendable API for classification, segmentation, detection (oriented bounding boxes), spectral indices, and geospatial operators. With our proposed framework, it is possible to control the results at three levels: (i) tool-level performance on public EO benchmarks; (ii) at the agent-level to understand the capacity to generate valid, hallucination-free code; and (iii) at the task-level on specific use cases. In this work, we select two use-cases of interest: land-composition mapping and post-wildfire damage assessment. The proposed agent outperforms general-purpose LLM/VLM baselines (GPT-4o, LLaVA), achieving 64.2% vs. 51.7% accuracy on land-composition and 50% vs. 0% on post-wildfire analysis, while producing results that are transparent and easy to interpret. By outputting verifiable code, the approach turns EO analysis into a transparent, reproducible process.

IC-EO: Interpretable Code-based assistant for Earth Observation

TL;DR

This work presents IC--EO, a code-first conversational agent that converts natural-language Earth Observation queries into executable Python workflows via a modular API of EO tools. By coupling GPT--4o with a structured tool registry and sandboxed execution, IC--EO delivers auditable, reproducible pipelines for tasks such as land-cover mapping and post-wildfire damage assessment, outperforming general-purpose VLM baselines on these use-cases. The evaluation spans tool-level, agent-level, and task-level metrics, highlighting high executable-code rates and transparent reasoning, while identifying memory and toolchain constraints as the main sources of failure. Overall, the study demonstrates that structured, code-generated EO workflows can achieve greater interpretability and reliability than end-to-end multimodal models, with strong potential for extensibility as new tools and datasets are added.

Abstract

Despite recent advances in computer vision, Earth Observation (EO) analysis remains difficult to perform for the laymen, requiring expert knowledge and technical capabilities. Furthermore, many systems return black-box predictions that are difficult to audit or reproduce. Leveraging recent advances in tool LLMs, this study proposes a conversational, code-generating agent that transforms natural-language queries into executable, auditable Python workflows. The agent operates over a unified easily extendable API for classification, segmentation, detection (oriented bounding boxes), spectral indices, and geospatial operators. With our proposed framework, it is possible to control the results at three levels: (i) tool-level performance on public EO benchmarks; (ii) at the agent-level to understand the capacity to generate valid, hallucination-free code; and (iii) at the task-level on specific use cases. In this work, we select two use-cases of interest: land-composition mapping and post-wildfire damage assessment. The proposed agent outperforms general-purpose LLM/VLM baselines (GPT-4o, LLaVA), achieving 64.2% vs. 51.7% accuracy on land-composition and 50% vs. 0% on post-wildfire analysis, while producing results that are transparent and easy to interpret. By outputting verifiable code, the approach turns EO analysis into a transparent, reproducible process.
Paper Structure (13 sections, 2 figures, 3 tables)

This paper contains 13 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. API Design Principles
  5. LLM-based code generation
  6. Experiments
  7. Definition of the API
  8. Definition of the Controller
  9. Execution environment and guardrails
  10. Evaluation protocol
  11. Results
  12. Conclusion
  13. Acknowledgements

Figures (2)

  • Figure 1: IC--EO framework overview From a user's natural language query, optionally with additional input data (such as images), the IC-EO framework produces a freely defined output, e.g. images or natural language. The user's query is first transformed with a LLM to an interpretable Python code which leverages an API or classical and recent EO models. This code can be executed to obtain the answer to the query.
  • Figure 2: Two short runs of IC--EO.