3D Optimization for AI Inference Scaling: Balancing Accuracy, Cost, and Latency

TL;DR

A 3D optimization framework that jointly calibrates accuracy, cost, and latency within a unified decision space, enabling constraints-aware inference scaling and shows that smaller models, when combined with optimal inference scaling, can match or exceed the performance of larger models at a fraction of the cost.

Abstract

AI inference scaling is often tuned through 1D heuristics (a fixed reasoning pass) or 2D bivariate trade-offs (e.g., accuracy vs. compute), which fail to consider cost and latency constraints. We introduce a 3D optimization framework that jointly calibrates accuracy, cost, and latency within a unified decision space, enabling constraints-aware inference scaling. Using Monte Carlo simulations across three representative scenarios and nine simulated large language models, we evaluate four optimization methods to address the 3D multi-objective optimization (MOO) problem. Framing inference scaling in MOO shapes a feasible space that 1D and 2D optimizations fail to capture, enabling environment-adaptive selection of the inference scaling~$k$. Results show that knee-point optimization based on Pareto frontiers achieves the best balance, while accuracy-maximization remains favorable when accuracy is prioritized. Our results further show that smaller models, when combined with optimal inference scaling, can match or exceed the performance of larger models at a fraction of the cost. The framework establishes a theoretical foundation for deployment-aware inference scaling across diverse operational conditions.

Figures (21)

• Figure 1: AI inference scaling requires optimizing the multidimensional tradeoff among cost, latency, and accuracy rather than relying on a bivariate performance-compute tradeoff that neglects the cost and latency constraints inherent in real-world deployment.
• Figure 2: Unlike conventional inference, which performs a single stochastic forward pass ($k{=}1$) to generate one output, parallel inference scaling executes multiple inference batches concurrently ($k \ge 2$) and aggregates their outputs (e.g., best-of-$k$) to improve accuracy with minimal increase in latency. Parallelism is particularly effective when sufficient computational resources are available to support concurrent execution.
• Figure 3: Compute-optimal AI system requires scaling efficiency in both training and inference. Training scaling laws (e.g., Chinchilla law) guide how models are built, while inference scaling laws govern how they are used efficiently under real-world cost and latency constraints.
• Figure 4: Inference-scaling optimization results for GPT-5 across three simulated scenarios. (a--c) 3D feasible cubes in cost--time--accuracy space with constraint planes of maximum cost, maximum latency, and minimal ACC ($C_{\max}$, $T_{\max}$, and $A_{\min}$). Markers denote each optimal point ($\bullet$ Maximum-ACC, $\blacktriangle$ Cube-Optimal, $\blacklozenge$ Utopia-Closest, $\blacksquare$ Knee-Point); purple colors indicate cube volume (larger is better), and the yellow star marks the utopia point.
• Figure 5: Accuracy-inference scale (k) trade-offs across three scenarios for GPT-5. These show MC means with CI = 95%; the red dashed line indicates minimal ACC ($A_{\min}$). Markers correspond to the optimal configurations identified in Fig. \ref{['fig:3d_results']}. Distinct operational priorities yield different optimal inference scales $k^\star$.