Evaluating Retrieval-Augmented Generation Variants for Natural Language-Based SQL and API Call Generation

TL;DR

This study assesses retrieval-augmented generation variants for translating natural language into SQL queries and REST API calls in an enterprise setting. It introduces a SAP Transactional Banking–based dataset and compares standard RAG, Self-RAG, and CoRAG across SQL, API, and a combined task under database-only, API-only, and hybrid documentation. Key findings show that retrieval is essential for meaningful accuracy, with CoRAG delivering statistically significant robustness in hybrid documentation, particularly for SQL generation within the combined task. The results underscore the importance of retrieval-policy design for production viability, advocate execution-based validation, and point to future directions like multi-turn interactions and cross-domain validation to enhance enterprise NL interfaces.

Abstract

Enterprise systems increasingly require natural language interfaces that can translate user requests into structured operations such as SQL queries and REST API calls. While large language models (LLMs) show promise for code generation [Chen et al., 2021; Huynh and Lin, 2025], their effectiveness in domain-specific enterprise contexts remains underexplored, particularly when both retrieval and modification tasks must be handled jointly. This paper presents a comprehensive evaluation of three retrieval-augmented generation (RAG) variants [Lewis et al., 2021] -- standard RAG, Self-RAG [Asai et al., 2024], and CoRAG [Wang et al., 2025] -- across SQL query generation, REST API call generation, and a combined task requiring dynamic task classification. Using SAP Transactional Banking as a realistic enterprise use case, we construct a novel test dataset covering both modalities and evaluate 18 experimental configurations under database-only, API-only, and hybrid documentation contexts. Results demonstrate that RAG is essential: Without retrieval, exact match accuracy is 0% across all tasks, whereas retrieval yields substantial gains in execution accuracy (up to 79.30%) and component match accuracy (up to 78.86%). Critically, CoRAG proves most robust in hybrid documentation settings, achieving statistically significant improvements in the combined task (10.29% exact match vs. 7.45% for standard RAG), driven primarily by superior SQL generation performance (15.32% vs. 11.56%). Our findings establish retrieval-policy design as a key determinant of production-grade natural language interfaces, showing that iterative query decomposition outperforms both top-k retrieval and binary relevance filtering under documentation heterogeneity.