Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text-to-OverpassQL: A Natural Language Interface for Complex Geodata Querying of OpenStreetMap

Michael Staniek, Raphael Schumann, Maike Züfle, Stefan Riezler

TL;DR

A detailed evaluation of the Text-to-OverpassQL task reveals strengths and weaknesses of the considered learning strategies, laying the foundations for further research into the Text-to-OverpassQL task.

Abstract

We present Text-to-OverpassQL, a task designed to facilitate a natural language interface for querying geodata from OpenStreetMap (OSM). The Overpass Query Language (OverpassQL) allows users to formulate complex database queries and is widely adopted in the OSM ecosystem. Generating Overpass queries from natural language input serves multiple use-cases. It enables novice users to utilize OverpassQL without prior knowledge, assists experienced users with crafting advanced queries, and enables tool-augmented large language models to access information stored in the OSM database. In order to assess the performance of current sequence generation models on this task, we propose OverpassNL, a dataset of 8,352 queries with corresponding natural language inputs. We further introduce task specific evaluation metrics and ground the evaluation of the Text-to-OverpassQL task by executing the queries against the OSM database. We establish strong baselines by finetuning sequence-to-sequence models and adapting large language models with in-context examples. The detailed evaluation reveals strengths and weaknesses of the considered learning strategies, laying the foundations for further research into the Text-to-OverpassQL task.

Text-to-OverpassQL: A Natural Language Interface for Complex Geodata Querying of OpenStreetMap

TL;DR

A detailed evaluation of the Text-to-OverpassQL task reveals strengths and weaknesses of the considered learning strategies, laying the foundations for further research into the Text-to-OverpassQL task.

Abstract

We present Text-to-OverpassQL, a task designed to facilitate a natural language interface for querying geodata from OpenStreetMap (OSM). The Overpass Query Language (OverpassQL) allows users to formulate complex database queries and is widely adopted in the OSM ecosystem. Generating Overpass queries from natural language input serves multiple use-cases. It enables novice users to utilize OverpassQL without prior knowledge, assists experienced users with crafting advanced queries, and enables tool-augmented large language models to access information stored in the OSM database. In order to assess the performance of current sequence generation models on this task, we propose OverpassNL, a dataset of 8,352 queries with corresponding natural language inputs. We further introduce task specific evaluation metrics and ground the evaluation of the Text-to-OverpassQL task by executing the queries against the OSM database. We establish strong baselines by finetuning sequence-to-sequence models and adapting large language models with in-context examples. The detailed evaluation reveals strengths and weaknesses of the considered learning strategies, laying the foundations for further research into the Text-to-OverpassQL task.
Paper Structure (32 sections, 2 equations, 14 figures, 4 tables)

This paper contains 32 sections, 2 equations, 14 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. OpenStreetMap
  4. Overpass Query Language
  5. Query Syntax
  6. Related Work
  7. Natural Language Interfaces for Geodata
  8. Text-to-SQL
  9. OverpassNL Dataset
  10. Query Annotation
  11. Dataset Statistics
  12. Complexity & Coverage
  13. Comparison to Other Datasets
  14. Task & Evaluation
  15. Overpass Query Similarity Evaluation
  16. ...and 17 more sections

Figures (14)

  • Figure 1: Natural language input and the corresponding Overpass query. The query is executed against the OpenStreetMap database and returns the requested elements in a structured response. The Overpass query language is highly expressive and allows to formulate complex queries to extract information from OpenStreetMap. Blue tokens in the query are syntax keywords, orange tokens are variable names and bold tokens define semantic properties of the requested elements. The green token in curly brackets geolocates an area called "Troms".
  • Figure 2: Dataset Statistics. Number of elements returned when executing the queries in the development set against OpenStreetMap. Each query returns at least one element and often several orders of magnitude more.
  • Figure 3: Location distribution of results returned by Overpass queries in our dataset. The queries cover locations on all continents. Europe is a traditional hotspot of the OpenStreetMap community and also has the best mapping coverage.
  • Figure 4: Development set results of LLaMa models with increasing number of parameters, prompted with five in-context examples.
  • Figure 5: Instance difficulty on the development set using OverpassT5. Dividing the evaluation instances by highest similarity to any training query allows to measure performance on instances with different difficulties.
  • ...and 9 more figures