Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguity

TL;DR

This work introduces BIRD-Python to benchmark Text-to-Python against Text-to-SQL on file-based data, demonstrating that Python can match SQL in core data retrieval when latent domain knowledge is explicitly provided. The authors show that the primary bottleneck is ambiguity in user intent rather than the code generation itself, and they propose the Logic Completion Framework (LCF) to inject clarifying knowledge and transform tasks into executable specifications. Through dataset corrections, SQL-to-Python reconstruction, semantic validation, and rigorous evaluation, the study reveals that with proper grounding, Text-to-Python achieves parity with Text-to-SQL across multiple model scales. The findings advocate grounding ambiguous natural language inputs in explicit logical specifications to unlock Python as a robust foundation for analytical agents operating on file-based data.

Abstract

While Text-to-SQL remains the dominant approach for database interaction, real-world analytics increasingly require the flexibility of general-purpose programming languages such as Python or Pandas to manage file-based data and complex analytical workflows. Despite this growing need, the reliability of Text-to-Python in core data retrieval remains underexplored relative to the mature SQL ecosystem. To address this gap, we introduce BIRD-Python, a benchmark designed for cross-paradigm evaluation. We systematically refined the original dataset to reduce annotation noise and align execution semantics, thereby establishing a consistent and standardized baseline for comparison. Our analysis reveals a fundamental paradigmatic divergence: whereas SQL leverages implicit DBMS behaviors through its declarative structure, Python requires explicit procedural logic, making it highly sensitive to underspecified user intent. To mitigate this challenge, we propose the Logic Completion Framework (LCF), which resolves ambiguity by incorporating latent domain knowledge into the generation process. Experimental results show that (1) performance differences primarily stem from missing domain context rather than inherent limitations in code generation, and (2) when these gaps are addressed, Text-to-Python achieves performance parity with Text-to-SQL. These findings establish Python as a viable foundation for analytical agents-provided that systems effectively ground ambiguous natural language inputs in executable logical specifications. Resources are available at https://anonymous.4open.science/r/Bird-Python-43B7/.

Figures (7)

• Figure 1: Functional comparison of SQL-based and code-based approaches across key data processing stages. SQL handles structured data and basic queries; code supports these tasks and adds flexibility—direct raw data processing, complex logic, machine learning integration, and automated dynamic output.
• Figure 2: Comparative framework of NLI semantic parsing paradigms. Top: Text-to-SQL maps queries to declarative relational algebra, delegating execution optimization to the DBMS. Bottom: Text-to-Python generates explicit procedural workflows, requiring the model to handle computational reasoning directly.
• Figure 3: Distribution of difference categories for verified SQL queries.
• Figure 4: Distribution of difference categories in SQL-to-Python conversion.
• Figure 5: The Logic Completion Framework (LCF). Standard semantic parsing models the probability of a program $P$ as $P(P | Q, \mathcal{S}, \mathcal{K})$. LCF makes latent domain knowledge explicit by providing logic clarifications ($C_{logic}$), refining the task to $P(P | Q, \mathcal{S}, \mathcal{K} \cup C_{logic})$.