Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FloodSQL-Bench: A Retrieval-Augmented Benchmark for Geospatially-Grounded Text-to-SQL

Hanzhou Liu, Kai Yin, Zhitong Chen, Chenyue Liu, Ali Mostafavi

TL;DR

FloodSQL-Bench addresses the lack of domain-specific, geospatially grounded Text-to-SQL benchmarks by introducing a real-world flood-management dataset suite with multi-table and geospatial reasoning. The benchmark combines ten heterogeneous tables spanning non-spatial, polygon, and point layers across TX, FL, and LA, and includes 443 question–SQL pairs organized by join complexity, enabling systematic evaluation under a retrieval-augmented generation framework with a rich metadata schema. The authors evaluate a wide range of LLMs, reveal that retrieval grounding improves complex spatial reasoning, and show performance gaps on the most challenging queries, highlighting the need for domain-aware, metadata-driven methods. The work provides an open, practical testbed for advancing geospatial Text-to-SQL research and disaster-management applications.

Abstract

Existing Text-to-SQL benchmarks primarily focus on single-table queries or limited joins in general-purpose domains, and thus fail to reflect the complexity of domain-specific, multi-table and geospatial reasoning, To address this limitation, we introduce FLOODSQL-BENCH, a geospatially grounded benchmark for the flood management domain that integrates heterogeneous datasets through key-based, spatial, and hybrid joins. The benchmark captures realistic flood-related information needs by combining social, infrastructural, and hazard data layers. We systematically evaluate recent large language models with the same retrieval-augmented generation settings and measure their performance across difficulty tiers. By providing a unified, open benchmark grounded in real-world disaster management data, FLOODSQL-BENCH establishes a practical testbed for advancing Text-to-SQL research in high-stakes application domains.

FloodSQL-Bench: A Retrieval-Augmented Benchmark for Geospatially-Grounded Text-to-SQL

TL;DR

FloodSQL-Bench addresses the lack of domain-specific, geospatially grounded Text-to-SQL benchmarks by introducing a real-world flood-management dataset suite with multi-table and geospatial reasoning. The benchmark combines ten heterogeneous tables spanning non-spatial, polygon, and point layers across TX, FL, and LA, and includes 443 question–SQL pairs organized by join complexity, enabling systematic evaluation under a retrieval-augmented generation framework with a rich metadata schema. The authors evaluate a wide range of LLMs, reveal that retrieval grounding improves complex spatial reasoning, and show performance gaps on the most challenging queries, highlighting the need for domain-aware, metadata-driven methods. The work provides an open, practical testbed for advancing geospatial Text-to-SQL research and disaster-management applications.

Abstract

Existing Text-to-SQL benchmarks primarily focus on single-table queries or limited joins in general-purpose domains, and thus fail to reflect the complexity of domain-specific, multi-table and geospatial reasoning, To address this limitation, we introduce FLOODSQL-BENCH, a geospatially grounded benchmark for the flood management domain that integrates heterogeneous datasets through key-based, spatial, and hybrid joins. The benchmark captures realistic flood-related information needs by combining social, infrastructural, and hazard data layers. We systematically evaluate recent large language models with the same retrieval-augmented generation settings and measure their performance across difficulty tiers. By providing a unified, open benchmark grounded in real-world disaster management data, FLOODSQL-BENCH establishes a practical testbed for advancing Text-to-SQL research in high-stakes application domains.

Paper Structure

This paper contains 26 sections, 2 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmark Construction
  4. Tabular Data Foundations
  5. Geographic Identifiers.
  6. Spatial representation.
  7. Join Semantics and Relations
  8. Metadata Builder
  9. Question and SQL Annotation
  10. Question Annotation.
  11. SQL Annotation.
  12. Experiments
  13. Evaluation Protocol
  14. RAG Framework
  15. (1) Table catalog.
  16. ...and 11 more sections

Figures (2)

  • Figure 1: A simplified overview of the annotation framework of our proposed FloodSQL-Bench.
  • Figure 2: The Retrieval-Augmented Generation (RAG) architecture used to evaluate different LLM agents on FloodSQL-Bench. We compute text embeddings for both table-level and column-level metadata and measure their cosine similarity with the embedded question (user query) sequentially. The top-scoring tables and columns are selected as retrieved candidates for downstream SQL generation.