Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An LLM Agent for Automatic Geospatial Data Analysis

Yuxing Chen, Weijie Wang, Sylvain Lobry, Camille Kurtz

TL;DR

The proposed GeoAgent pioneers the integration of a code interpreter, static analysis, and Retrieval-Augmented Generation techniques within a Monte Carlo Tree Search algorithm, offering a novel approach to geospatial data processing.

Abstract

Large language models (LLMs) are being used in data science code generation tasks, but they often struggle with complex sequential tasks, leading to logical errors. Their application to geospatial data processing is particularly challenging due to difficulties in incorporating complex data structures and spatial constraints, effectively utilizing diverse function calls, and the tendency to hallucinate less-used geospatial libraries. To tackle these problems, we introduce GeoAgent, a new interactive framework designed to help LLMs handle geospatial data processing more effectively. GeoAgent pioneers the integration of a code interpreter, static analysis, and Retrieval-Augmented Generation (RAG) techniques within a Monte Carlo Tree Search (MCTS) algorithm, offering a novel approach to geospatial data processing. In addition, we contribute a new benchmark specifically designed to evaluate the LLM-based approach in geospatial tasks. This benchmark leverages a variety of Python libraries and includes both single-turn and multi-turn tasks such as data acquisition, data analysis, and visualization. By offering a comprehensive evaluation among diverse geospatial contexts, this benchmark sets a new standard for developing LLM-based approaches in geospatial data analysis tasks. Our findings suggest that relying solely on knowledge of LLM is insufficient for accurate geospatial task programming, which requires coherent multi-step processes and multiple function calls. Compared to the baseline LLMs, the proposed GeoAgent has demonstrated superior performance, yielding notable improvements in function calls and task completion. In addition, these results offer valuable insights for the future development of LLM agents in automatic geospatial data analysis task programming.

An LLM Agent for Automatic Geospatial Data Analysis

TL;DR

The proposed GeoAgent pioneers the integration of a code interpreter, static analysis, and Retrieval-Augmented Generation techniques within a Monte Carlo Tree Search algorithm, offering a novel approach to geospatial data processing.

Abstract

Large language models (LLMs) are being used in data science code generation tasks, but they often struggle with complex sequential tasks, leading to logical errors. Their application to geospatial data processing is particularly challenging due to difficulties in incorporating complex data structures and spatial constraints, effectively utilizing diverse function calls, and the tendency to hallucinate less-used geospatial libraries. To tackle these problems, we introduce GeoAgent, a new interactive framework designed to help LLMs handle geospatial data processing more effectively. GeoAgent pioneers the integration of a code interpreter, static analysis, and Retrieval-Augmented Generation (RAG) techniques within a Monte Carlo Tree Search (MCTS) algorithm, offering a novel approach to geospatial data processing. In addition, we contribute a new benchmark specifically designed to evaluate the LLM-based approach in geospatial tasks. This benchmark leverages a variety of Python libraries and includes both single-turn and multi-turn tasks such as data acquisition, data analysis, and visualization. By offering a comprehensive evaluation among diverse geospatial contexts, this benchmark sets a new standard for developing LLM-based approaches in geospatial data analysis tasks. Our findings suggest that relying solely on knowledge of LLM is insufficient for accurate geospatial task programming, which requires coherent multi-step processes and multiple function calls. Compared to the baseline LLMs, the proposed GeoAgent has demonstrated superior performance, yielding notable improvements in function calls and task completion. In addition, these results offer valuable insights for the future development of LLM agents in automatic geospatial data analysis task programming.

Paper Structure

This paper contains 31 sections, 8 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. LLM-based Code Generation
  4. Geospatial Analysis with Large Language Models
  5. Methodology
  6. Framework
  7. API Retrieval
  8. Dynamic Refinement on MCTS
  9. Experimental Setup
  10. Benchmark Construction
  11. Benchmark Statistics
  12. RAG Library
  13. Experimental Settings
  14. Metrics
  15. Evaluation
  16. ...and 16 more sections

Figures (4)

  • Figure 1: GeoAgent: A geospatial data analysis task programming agent. This agent comprises four integral components: b) Retriever, which retrieves task-relevant items from provided document data, including Python library documents, online tutorials, and function-level solutions; c) Action, which is an execution environment integrated with a code interpreter and static analysis, provides feedback on generated code and suggests potential fixes; d) MCTS, explores and evaluates multiple possible code candidates to optimize the selection of the most promising solution at each step through iterative adjustment and refinement; and LLMs, function as both the code generator and the intelligent reasoner to propose code solutions and iteratively diagnose and fix errors.
  • Figure 2: Illustration of using the Monte Carlo tree search algorithm in geospatial data analysis task programming.
  • Figure 3: The self-refinement algorithm in MCTS.
  • Figure 4: Addition of new tasks to the task pool in MCTS.