HEXGEN-FLOW: Optimizing LLM Inference Request Scheduling for Agentic Text-to-SQL

TL;DR

HEXGEN-FLOW addresses the challenge of efficiently serving agentic, multi-stage Text-to-SQL workloads on heterogeneous GPU clusters while meeting end-to-end SLOs in multi-tenant environments. It introduces a two-level scheduling framework—global workload-balanced dispatching and per-model adaptive urgency queues—complemented by a simulation-guided α-tuning mechanism to adapt to changing workloads. By explicitly modeling inter-stage dependencies and hardware heterogeneity, the approach achieves substantial reductions in P95 tail latency and notable throughput gains across diverse traces and hardware configurations. The results demonstrate robust performance improvements over state-of-the-art LLM serving baselines, enabling more reliable production deployment of agentic Text-to-SQL systems. This work advances practical LLM serving by aligning end-to-end latency guarantees with real-world multi-tenant, heterogeneous data-center environments.

Abstract

Recent advancements in leveraging the agentic paradigm of large language models (LLMs) have substantially improved Text-to-SQL capabilities, empowering users without specialized database knowledge to intuitively query databases. However, deploying agentic LLM-based Text-to-SQL systems in production presents significant challenges, stemming from their inherently multi-stage computational dependencies, strict latency requirements, and the complexity of deployment across heterogeneous GPUs widely existing in enterprise clusters. Meanwhile, existing LLM serving frameworks are primarily designed for independent inference tasks, resulting in suboptimal performance and frequent service-level objective (SLO) violations in Text-to-SQL workloads. In this paper, we introduce HEXGEN-FLOW, a novel framework designed explicitly to schedule and execute agentic multi-stage LLM-based Text-to-SQL workflows on heterogeneous GPU clusters serving multi-tenant Text-to-SQL requests. HEXGEN-FLOW introduces a hierarchical scheduling approach that combines global workload-balanced task dispatching with an adaptive local priority queue, guided by a systematic analysis of agentic Text-to-SQL workflows. Additionally, we propose a lightweight simulation-based method for tuning critical scheduling hyperparameters, further enhancing robustness and adaptability. Our evaluation on realistic Text-to-SQL benchmarks demonstrates that HEXGEN-FLOW significantly outperforms state-of-the-art LLM serving frameworks. Across all traces, HEXGEN-FLOW reduces P95 tail latency by $1.42{\sim}1.56\times$ and increases throughput by $1.49{\sim}1.81\times$, demonstrating robust improvements under diverse workloads. Our code is available at https://github.com/Relaxed-System-Lab/Hexgen-Flow.

Figures (10)

• Figure 1: A visual illustration of the agentic multiple-stage Text-to-SQL workflow with inter-stage dependency.
• Figure 2: Hexgen-Flow system. Incoming LLM inference requests are dispatched by a global coordinator to model instances based on workload balance and task suitability. Each model instance manages its queue using an urgency-guided priority mechanism. Different color chunks represent an LLM inference request from a different stage in the workflow.
• Figure 3: The SLO budget assigned to $q_{i,1}$ is $40$ms, but $q_{i,1}$ takes $50$ms to finish. The additional 10ms debit is recorded by $\tau^{i}_{\text{elapsed}}$ and reallocated to the following inference requests, resulting in a tighter SLO budget for $q_{i,2}$.
• Figure 4: End-to-end SLO attainment comparison. $r$ represents the request rate in requests per second.
• Figure 5: End-to-end system throughput comparison.