Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semantic Caching and Intent-Driven Context Optimization for Multi-Agent Natural Language to Code Systems

Harmohit Singh

TL;DR

This work presents a production-ready, multi-agent NL2Code system for enterprise analytics that translates natural language queries into Python code while controlling cost and latency. Core innovations include a semantic cache with LLM-based equivalence and explicit field adaptations, a five-level QuerySignature for robust cache matching, a dual-threshold decision mechanism separating exact-match and guided generation, and an intent-driven dynamic prompt assembly that cuts token usage by up to 60%. In production, the approach achieves 94.3% semantic accuracy with 8.2s average latency and 67.3% total cache utilization across 10k+ queries, using cost-efficient models. The architecture combines specialized agents, structured adaptation hints, and prompt-driven optimization to offer scalable, explainable NL2Code for enterprise data workflows. These results demonstrate practical viability and provide a blueprint for deploying LLM-based analytics at scale, along with considerations for deployment, limitations, and ethical safeguards.

Abstract

We present a production-optimized multi-agent system designed to translate natural language queries into executable Python code for structured data analytics. Unlike systems that rely on expensive frontier models, our approach achieves high accuracy and cost efficiency through three key innovations: (1) a semantic caching system with LLM-based equivalence detection and structured adaptation hints that provides cache hit rates of 67% on production queries; (2) a dual-threshold decision mechanism that separates exact-match retrieval from reference-guided generation; and (3) an intent-driven dynamic prompt assembly system that reduces token consumption by 40-60% through table-aware context filtering. The system has been deployed in production for enterprise inventory management, processing over 10,000 queries with an average latency of 8.2 seconds and 94.3% semantic accuracy. We describe the architecture, present empirical results from production deployment, and discuss practical considerations for deploying LLM-based analytics systems at scale.

Semantic Caching and Intent-Driven Context Optimization for Multi-Agent Natural Language to Code Systems

TL;DR

This work presents a production-ready, multi-agent NL2Code system for enterprise analytics that translates natural language queries into Python code while controlling cost and latency. Core innovations include a semantic cache with LLM-based equivalence and explicit field adaptations, a five-level QuerySignature for robust cache matching, a dual-threshold decision mechanism separating exact-match and guided generation, and an intent-driven dynamic prompt assembly that cuts token usage by up to 60%. In production, the approach achieves 94.3% semantic accuracy with 8.2s average latency and 67.3% total cache utilization across 10k+ queries, using cost-efficient models. The architecture combines specialized agents, structured adaptation hints, and prompt-driven optimization to offer scalable, explainable NL2Code for enterprise data workflows. These results demonstrate practical viability and provide a blueprint for deploying LLM-based analytics at scale, along with considerations for deployment, limitations, and ethical safeguards.

Abstract

We present a production-optimized multi-agent system designed to translate natural language queries into executable Python code for structured data analytics. Unlike systems that rely on expensive frontier models, our approach achieves high accuracy and cost efficiency through three key innovations: (1) a semantic caching system with LLM-based equivalence detection and structured adaptation hints that provides cache hit rates of 67% on production queries; (2) a dual-threshold decision mechanism that separates exact-match retrieval from reference-guided generation; and (3) an intent-driven dynamic prompt assembly system that reduces token consumption by 40-60% through table-aware context filtering. The system has been deployed in production for enterprise inventory management, processing over 10,000 queries with an average latency of 8.2 seconds and 94.3% semantic accuracy. We describe the architecture, present empirical results from production deployment, and discuss practical considerations for deploying LLM-based analytics systems at scale.
Paper Structure (55 sections, 3 equations, 2 figures, 9 tables)

This paper contains 55 sections, 3 equations, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Motivation and Problem Statement
  3. Contributions
  4. Paper Organization
  5. Related Work
  6. Natural Language to Code Systems
  7. Text-to-SQL and Enterprise Benchmarks
  8. Semantic Caching for LLMs
  9. Multi-Agent LLM Systems
  10. Summary and Positioning
  11. System Architecture
  12. Agent Descriptions
  13. Guard Agent ($\mathcal{G}$)
  14. Intent Classifier ($\mathcal{I}$)
  15. Reference Matcher ($\mathcal{R}$)
  16. ...and 40 more sections

Figures (2)

  • Figure 1: Multi-Agent Architecture with Semantic Caching. The system processes queries through preprocessing, cache lookup with dual-threshold decision ($\theta_R = 0.995$, $\theta_G = 0.50$), conditional generation pipeline, and output processing. Dashed arrows indicate adaptation hints $\Delta$ propagated from cached references. Mathematical notation: $\mathcal{G}$ = Guard function, $\mathcal{I}$ = Intent classifier, $\mathcal{R}$ = Reference matcher, $\mathcal{C}$ = Cache store, $\mathcal{P}$ = Planner, $\mathcal{A}$ = Code generator, $\mathcal{E}$ = Executor, $\mathcal{S}$ = Summarizer, $\mathcal{B}$ = Business insights generator.
  • Figure 2: Distribution of embedding similarity scores across production queries. 67.3% of queries have similarity $\geq 0.50$, qualifying for cache utilization.