TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQL

TL;DR

This work tackles the latency bottleneck in Text-to-SQL inference caused by large schema prompts by offline precomputing per-table KV caches that preserve primary–foreign key relationships. It introduces a Primary Foreign Key Guided Table Representation and a Table Trie to enable efficient online lookup and cache assembly, complemented by a cache-management framework with query reranking and a computation loading pipeline to hide memory latency. Empirical results show TableCache achieving up to $3.62\times$ TTFT speedups with negligible accuracy loss across Spider and BIRD benchmarks and backbones like OmniSQL-7B and Qwen2.5-7B-coder, with strong ablations validating the importance of PFTR, reranking, and the loading pipeline. The approach offers practical deployment benefits in enterprise Text-to-SQL systems by reducing prefill latency while maintaining performance, and presents a scalable pathway for integrating schema-aware caching into existing KV-cache ecosystems.

Abstract

In Text-to-SQL tasks, existing LLM-based methods often include extensive database schemas in prompts, leading to long context lengths and increased prefilling latency. While user queries typically focus on recurrent table sets-offering an opportunity for KV cache sharing across queries-current inference engines, such as SGLang and vLLM, generate redundant prefix cache copies when processing user queries with varying table orders. To address this inefficiency, we propose precomputing table representations as KV caches offline and querying the required ones online. A key aspect of our approach is the computation of table caches while preserving primary foreign key relationships between tables. Additionally, we construct a Table Trie structure to facilitate efficient KV cache lookups during inference. To enhance cache performance, we introduce a cache management system with a query reranking strategy to improve cache hit rates and a computation loading pipeline for parallelizing model inference and cache loading. Experimental results show that our proposed TableCache achieves up to a 3.62x speedup in Time to First Token (TTFT) with negligible performance degradation.

Figures (3)

• Figure 1: An example of TableCache
• Figure 2: Overview of TableCache. KV caches are precomputed and organized into a hierarchical Table Trie during offline preprocessing. Inference utilizes cache management system, including query reranking for cache-friendly sequencing and a computation loading pipeline to reduce latency and improve efficiency.
• Figure 3: Primary Foreign Key Guided Table Representation.