Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Declarative Techniques for NL Queries over Heterogeneous Data

Elham Khabiri, Jeffrey O. Kephart, Fenno F. Heath, Srideepika Jayaraman, Fateh A. Tipu, Yingjie Li, Dhruv Shah, Achille Fokoue, Anu Bhamidipaty

TL;DR

This paper tackles natural language querying in industrial environments where data lives across heterogeneous sources (databases, APIs, etc.). It proposes siwarex, a declarative framework that represents user intent with SQL and integrates external APIs via User Defined Functions, enabling seamless orchestration alongside databases. Two Spider-based benchmarks augmented with API calls are introduced to evaluate performance in mixed-source scenarios, with results showing the declarative approach outperforms imperative and agent-based methods, especially as data heterogeneity increases. The work provides open benchmarks and outlines practical considerations for real-world deployment, while identifying limitations and future work in handling larger schemas and vector/table outputs.

Abstract

In many industrial settings, users wish to ask questions in natural language, the answers to which require assembling information from diverse structured data sources. With the advent of Large Language Models (LLMs), applications can now translate natural language questions into a set of API calls or database calls, execute them, and combine the results into an appropriate natural language response. However, these applications remain impractical in realistic industrial settings because they do not cope with the data source heterogeneity that typifies such environments. In this work, we simulate the heterogeneity of real industry settings by introducing two extensions of the popular Spider benchmark dataset that require a combination of database and API calls. Then, we introduce a declarative approach to handling such data heterogeneity and demonstrate that it copes with data source heterogeneity significantly better than state-of-the-art LLM-based agentic or imperative code generation systems. Our augmented benchmarks are available to the research community.

Declarative Techniques for NL Queries over Heterogeneous Data

TL;DR

This paper tackles natural language querying in industrial environments where data lives across heterogeneous sources (databases, APIs, etc.). It proposes siwarex, a declarative framework that represents user intent with SQL and integrates external APIs via User Defined Functions, enabling seamless orchestration alongside databases. Two Spider-based benchmarks augmented with API calls are introduced to evaluate performance in mixed-source scenarios, with results showing the declarative approach outperforms imperative and agent-based methods, especially as data heterogeneity increases. The work provides open benchmarks and outlines practical considerations for real-world deployment, while identifying limitations and future work in handling larger schemas and vector/table outputs.

Abstract

In many industrial settings, users wish to ask questions in natural language, the answers to which require assembling information from diverse structured data sources. With the advent of Large Language Models (LLMs), applications can now translate natural language questions into a set of API calls or database calls, execute them, and combine the results into an appropriate natural language response. However, these applications remain impractical in realistic industrial settings because they do not cope with the data source heterogeneity that typifies such environments. In this work, we simulate the heterogeneity of real industry settings by introducing two extensions of the popular Spider benchmark dataset that require a combination of database and API calls. Then, we introduce a declarative approach to handling such data heterogeneity and demonstrate that it copes with data source heterogeneity significantly better than state-of-the-art LLM-based agentic or imperative code generation systems. Our augmented benchmarks are available to the research community.
Paper Structure (25 sections, 4 figures, 3 tables)

This paper contains 25 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Structured data retrieval
  4. LLM-based tool calling
  5. Handling heterogeneous data sources
  6. Approaches to handling heterogeneity
  7. Declarative approach
  8. Imperative approach
  9. Agentic approach
  10. Two benchmark datasets
  11. Benchmark I
  12. Benchmark II
  13. Experiments
  14. Benchmark I experiments
  15. Benchmark II experiments
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Example of schemas and table view used by siwarex. The Abstract Schema and API Mapping Schema required by siwarex can be provided manually or extracted from domain metadata. If a database schema is provided, the Abstract Schema can be extracted from it automatically; likewise the API Mapping Schema can be extracted automatically from an OpenAPI spec. For systems that mix DB access and API calls, the edges between API and DB nodes in the Abstract Schema may be augmented by a minimal amount of expert knowledge. Once the Abstract Schema is created, a relational schema (DB Table View) is generated from it automatically. The DB Table View represents all entities consistently as tables regardless of whether they are actually tables or APIs.
  • Figure 2: Measured accuracies for Agentic (xLAM-based), Imperative and Declarative methods vs the percentage of DB tables that were replaced by APIs.
  • Figure 3: Closeup screenshot of UI used to compare ground truth candidates generated by QR and SL methods, in the case where both produce rows and those rows are deemed mutually compatible.
  • Figure 4: Screenshot of UI used to compare ground truth candidates.