Semantic Captioning: Benchmark Dataset and Graph-Aware Few-Shot In-Context Learning for SQL2Text

TL;DR

This work introduces semantic captioning as the SQL2Text reverse task and develops iterative in-context learning (ICL) to generate robust natural language captions from SQL queries. It repurposes Text2SQL datasets (CoSQL, Spider, SParC) into a SQL2Text benchmark using an iterative prompt with GPT-4o and conducts rigorous automatic and human evaluations, achieving high semantic and logical alignment (AlignScore and $BERTScore$). To improve ICL effectiveness for smaller models, it proposes AST-ICL, a graph-aware sample selection method that encodes SQL as an AST, embeds it with a GNN, and clusters for demonstration retrieval; AST-ICL-TOP further selects top-similar samples. Results show graph-aware sampling yields substantial gains over random and BM25 baselines, with smaller LLMs approaching the performance of GPT-4 in many settings, highlighting practical avenues for efficient code understanding and documentation tools. The work also discusses dataset quality challenges and outlines future directions for broader language support, enhanced graph representations, and standardized evaluation pipelines.

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance in various NLP tasks, including semantic parsing, which translates natural language into formal code representations. However, the reverse process, translating code into natural language, termed semantic captioning, has received less attention. This task is becoming increasingly important as LLMs are integrated into platforms for code generation, security analysis, and educational purposes. In this paper, we focus on the captioning of SQL query (SQL2Text) to address the critical need for understanding and explaining SQL queries in an era where LLM-generated code poses potential security risks. We repurpose Text2SQL datasets for SQL2Text by introducing an iterative ICL prompt using GPT-4o to generate multiple additional utterances, which enhances the robustness of the datasets for the reverse task. We conduct our experiments using in-context learning (ICL) based on different sample selection methods, emphasizing smaller, more computationally efficient LLMs. Our findings demonstrate that leveraging the inherent graph properties of SQL for ICL sample selection significantly outperforms random selection by up to 39% on BLEU score and provides better results than alternative methods. Dataset and codes are published: https://github.com/aliwister/ast-icl.

Figures (4)

• Figure 1: Graph AST-ICL Framework for SQL2Text generation showing the pipeline from SQL input through Graph-AST methods and ICL approaches to final text output using various LLM generators (e.g., Mistral).
• Figure 2: Parse Tree for SQL Query: SELECT count(*) FROM Dogs WHERE dog_id NOT IN (SELECT dog_id FROM Treatments)
• Figure 3: AST-ICL Model Overview: First, AST graphs are constructed for each SQL segment, then encoded into vectors by a GNN embedding model, and finally clustered using k-means for AST-based In-Context Learning (ICL).
• Figure 4: The total time (s) to generate the prompt samples for ICL by method, and dataset. Smaller datasets demonstrate a shorter time