Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cost-Aware Text-to-SQL: An Empirical Study of Cloud Compute Costs for LLM-Generated Queries

Saurabh Deochake, Debajyoti Mukhopadhyay

TL;DR

This study introduces a cloud-native, cost-focused evaluation of LLM-generated Text-to-SQL on Google BigQuery using the StackOverflow dataset, revealing that reasoning-models reduce cloud compute costs by about $44.5\%$ compared with standard models while maintaining near-perfect correctness. It demonstrates a weak link between execution time and cost, showing that faster queries do not imply cheaper ones, and highlights substantial cost variance across models driven by inefficiency patterns such as missing partition filters and unnecessary full-table scans. The authors provide practical deployment guidelines and a framework for cost-aware benchmarking, emphasizing the need for explicit cloud-cost metrics in production. The work lays groundwork for cost-aware prompt design, model selection, and operational policies in enterprise settings for Text-to-SQL systems.

Abstract

Text-to-SQL systems powered by Large Language Models (LLMs) achieve high accuracy on standard benchmarks, yet existing efficiency metrics such as the Valid Efficiency Score (VES) measure execution time rather than the consumption-based costs of cloud data warehouses. This paper presents the first systematic evaluation of cloud compute costs for LLM-generated SQL queries. We evaluate six state-of-the-art LLMs across 180 query executions on Google BigQuery using the StackOverflow dataset (230GB), measuring bytes processed, slot utilization, and estimated cost. Our analysis yields three key findings: (1) reasoning models process 44.5% fewer bytes than standard models while maintaining equivalent correctness (96.7%-100%); (2) execution time correlates weakly with query cost (r=0.16), indicating that speed optimization does not imply cost optimization; and (3) models exhibit up to 3.4x cost variance, with standard models producing outliers exceeding 36GB per query. We identify prevalent inefficiency patterns including missing partition filters and unnecessary full-table scans, and provide deployment guidelines for cost-sensitive enterprise environments.

Cost-Aware Text-to-SQL: An Empirical Study of Cloud Compute Costs for LLM-Generated Queries

TL;DR

This study introduces a cloud-native, cost-focused evaluation of LLM-generated Text-to-SQL on Google BigQuery using the StackOverflow dataset, revealing that reasoning-models reduce cloud compute costs by about compared with standard models while maintaining near-perfect correctness. It demonstrates a weak link between execution time and cost, showing that faster queries do not imply cheaper ones, and highlights substantial cost variance across models driven by inefficiency patterns such as missing partition filters and unnecessary full-table scans. The authors provide practical deployment guidelines and a framework for cost-aware benchmarking, emphasizing the need for explicit cloud-cost metrics in production. The work lays groundwork for cost-aware prompt design, model selection, and operational policies in enterprise settings for Text-to-SQL systems.

Abstract

Text-to-SQL systems powered by Large Language Models (LLMs) achieve high accuracy on standard benchmarks, yet existing efficiency metrics such as the Valid Efficiency Score (VES) measure execution time rather than the consumption-based costs of cloud data warehouses. This paper presents the first systematic evaluation of cloud compute costs for LLM-generated SQL queries. We evaluate six state-of-the-art LLMs across 180 query executions on Google BigQuery using the StackOverflow dataset (230GB), measuring bytes processed, slot utilization, and estimated cost. Our analysis yields three key findings: (1) reasoning models process 44.5% fewer bytes than standard models while maintaining equivalent correctness (96.7%-100%); (2) execution time correlates weakly with query cost (r=0.16), indicating that speed optimization does not imply cost optimization; and (3) models exhibit up to 3.4x cost variance, with standard models producing outliers exceeding 36GB per query. We identify prevalent inefficiency patterns including missing partition filters and unnecessary full-table scans, and provide deployment guidelines for cost-sensitive enterprise environments.
Paper Structure (30 sections, 1 equation, 4 figures, 7 tables)

This paper contains 30 sections, 1 equation, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-SQL with Large Language Models
  4. Text-to-SQL Benchmarks
  5. Query Optimization and Cost Estimation
  6. Methodology
  7. Evaluation Platform
  8. Benchmark Workload
  9. Query Benchmark Design
  10. Large Language Models
  11. Prompt Design
  12. Cost Metrics
  13. Derived Metrics
  14. Experimental Procedure
  15. Results
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Mean bytes processed per query by model. Reasoning models (blue, superscript R) demonstrate higher efficiency than standard models (orange).
  • Figure 2: Cost disparities by query complexity. While costs are comparable for simple queries (16% difference), standard models exhibit significantly higher costs for complex queries (115% difference).
  • Figure 3: Distribution of bytes processed. Y-axis clipped at 14GB for readability; Gemini Flash and GPT-5.1 had extreme outliers (28.8GB and 36.6GB) exceeding the chart limits.
  • Figure 4: Scatter plot showing weak correlation ($r=0.16$) between bytes processed and execution time. Fast queries are not necessarily cheap queries in cloud data warehouses.