Statistical Modeling and Uncertainty Estimation of LLM Inference Systems

TL;DR

The paper tackles the problem of statistically characterizing LLM inference performance under dynamic workloads and heterogeneous hardware. It introduces Analytical with Learning Augmentation (ALA), a hybrid approach that couples a Generalized Exponential Throughput Formula, $thpt = c - a e^{-b bb}$, with ML-based parameter prediction, simulated annealing for selective training data usage, an error predictor, and uncertainty quantified through vector-space similarity. Key contributions include an Exponential Parameter Database, a multi-output XGBoost predictor for $(a,b,c)$, an SA-driven error model, and principled uncertainty estimates, demonstrating low median prediction errors and robust generalization across diverse workloads. The approach enables scalable workload scheduling, adaptive resource provisioning, and cost-aware inference optimization in large-scale LLM deployments, by providing accurate throughput and latency statistics for unseen configurations. This hybrid methodology offers practical impact for deploy-time planning and performance tuning in real-world AI inference systems.

Abstract

Large Language Model (LLM) inference systems present significant challenges in statistical performance characterization due to dynamic workload variations, diverse hardware architectures, and complex interactions between model size, batch processing, and throughput requirements. Accurate statistical characterization enables better workload scheduling, adaptive resource provisioning, and cost-aware inference optimization, making it crucial for improving efficiency in large-scale AI deployments. Traditional analytical models provide explainability but cannot cover the vast diversity of real-world workloads, making it impossible to benchmark every scenario in advance. Machine learning (ML) approaches effectively predict performance for non-benchmarked cases but struggle when extrapolating beyond their observed training space. To address these limitations for LLM inference systems, we propose an Analytical with Learning Augmentation (ALA) framework that bridges analytical modeling with \ml for robust statistical prediction and uncertainty estimation in LLM inference workloads. Our method employs an analytical throughput model with parameters estimated for benchmarked workloads, then extends to unobserved configurations using \ml predictions. We enhance this with simulated annealing to exploit subsets of the workload data point combinations and develop an error predictor. Finally, we quantify uncertainty based on vector space similarity between new and observed workloads to ensure robust generalization. Through extensive experimentation on diverse LLM inference workloads, we demonstrate that our framework achieves low median errors while maintaining adaptability to new inference scenarios.

Figures (8)

• Figure 1: ALA Framework Solution Overview
• Figure 2: Exponential Models for Throughput in LLAMA
• Figure 3: Exponential Models for Throughput in LLAMA
• Figure 4: Dataset Statistical Characteristics showing throughput variation with (a) batch size, (b) input token size, and (c) output token size.
• Figure 5: Distribution of Training Sets