AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving

TL;DR

AIConfigurator tackles the challenge of optimizing LLM inference in production by delivering a framework-agnostic, data-driven configuration search that does not require GPU profiling. It achieves this through an operator-level performance model that decomposes inference into GEMM, Attention, and communication primitives, backed by a calibrated kernel-level database and an abstraction layer that outputs optimal launch parameters. The method supports static, aggregated, and disaggregated serving modes, including a MoE-aware power-law correction, enabling fast, accurate end-to-end estimates across diverse frameworks (e.g., TensorRT-LLM, vLLM, SGLang). Evaluations show up to 40-50% performance gains and average search times of about 30 seconds on CPU, with strong fidelity to ground-truth measurements across dense and MoE models. This work significantly reduces the cost and time of deploying optimized LLM inference, facilitating rapid exploration of large design spaces from cluster topology to engine flags.”

Abstract

Optimizing Large Language Model (LLM) inference in production systems is increasingly difficult due to dynamic workloads, stringent latency/throughput targets, and a rapidly expanding configuration space. This complexity spans not only distributed parallelism strategies (tensor/pipeline/expert) but also intricate framework-specific runtime parameters such as those concerning the enablement of CUDA graphs, available KV-cache memory fractions, and maximum token capacity, which drastically impact performance. The diversity of modern inference frameworks (e.g., TRT-LLM, vLLM, SGLang), each employing distinct kernels and execution policies, makes manual tuning both framework-specific and computationally prohibitive. We present AIConfigurator, a unified performance-modeling system that enables rapid, framework-agnostic inference configuration search without requiring GPU-based profiling. AIConfigurator combines (1) a methodology that decomposes inference into analytically modelable primitives - GEMM, attention, communication, and memory operations while capturing framework-specific scheduling dynamics; (2) a calibrated kernel-level performance database for these primitives across a wide range of hardware platforms and popular open-weights models (GPT-OSS, Qwen, DeepSeek, LLama, Mistral); and (3) an abstraction layer that automatically resolves optimal launch parameters for the target backend, seamlessly integrating into production-grade orchestration systems. Evaluation on production LLM serving workloads demonstrates that AIConfigurator identifies superior serving configurations that improve performance by up to 40% for dense models (e.g., Qwen3-32B) and 50% for MoE architectures (e.g., DeepSeek-V3), while completing searches within 30 seconds on average. Enabling the rapid exploration of vast design spaces - from cluster topology down to engine specific flags.

Figures (8)

• Figure 1: AIConfigurator projected Throughput vs Speed Pareto frontiers for Qwen3-235B running on 64 H200 GPUs. All serving configurations that can achieve a TTFT (Time to First Token) $\leq$ 1000ms are plotted on the chart.
• Figure 2: Key components of AIConfigurator and the general workflow of finding the optimal configuration.
• Figure 3: Three serving modes modeled by AIConfigurator. (A) Static: GPU workers process fixed inference requests end-to-end. (B) Aggregated: prefill and decode of different requests are mixed (continuous batching). (C) Disaggregated: separate GPU pools for prefill and decode phases.
• Figure 4: A step of LLM inference can be decomposed into repeated execution of a few key operators. For instance, the inference step of a typical MoE model normally involves the above depicted operators, and how expert parallelism is implemented depends on the specific backend used.
• Figure 5: Visualizing the effect of $\alpha$. As $\alpha$ increases, the routing distribution shifts from perfectly balanced (uniform) to highly skewed, where the top-ranked experts (E1, E2) handle the majority of tokens.