From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models

TL;DR

The paper addresses the cost-performance tension in online generative-model inference by shifting autoscaling granularity from models to operators. It introduces a queueing-theory–driven framework and greedy algorithms for operator-level autoscaling and device placement, leveraging per-operator compute, memory, and communication profiles. Empirical evaluation on production-scale traces demonstrates substantial improvements, including up to $40\%$ fewer GPUs and $35\%$ less energy, with larger gains for prefill stages and larger models, while maintaining $TTFT$ and $TBT$ SLOs. The results show that operator granularity enables faster elasticity and tighter coupling of resource allocation to workload dynamics, offering a practical path to more efficient large-scale generative serving.

Abstract

Serving large generative models such as LLMs and multi- modal transformers requires balancing user-facing SLOs (e.g., time-to-first-token, time-between-tokens) with provider goals of efficiency and cost reduction. Existing solutions rely on static provisioning or model-level autoscaling, both of which treat the model as a monolith. This coarse-grained resource management leads to degraded performance or significant resource underutilization due to poor adaptability to dynamic inference traffic that is common online. The root cause of this inefficiency lies in the internal structure of generative models: they are executed as graphs of interconnected operators. Through detailed characterization and systematic analysis, we find that operators are heterogeneous in their compute and memory footprints and exhibit diverse sensitivity to workload and resource factors such as batch size, sequence length, and traffic rate. This heterogeneity suggests that the operator, rather than the entire model, is the right granularity for scaling decisions. We propose an operator-level autoscaling framework, which allocates resources at finer (operator)-granularity, optimizing the scaling, batching, and placement based on individual operator profiles. Evaluated on production-scale traces, our approach preserves SLOs with up to 40% fewer GPUs and 35% less energy, or under fixed resources achieves 1.6x higher throughput with 5% less energy. These results show that the operator, rather than the model, is fundamentally a more effective unit for scaling large generative workloads.

Figures (13)

• Figure 1: Operator-level vs. model-level autoscaling.
• Figure 2: Compute sensitivity to input data sizes, for various operators in different model architectures.
• Figure 3: Memory sensitivity to input data size, for various operators in different model architectures.
• Figure 4: Compute sensitivity to batch sizes.
• Figure 5: Compute- vs. memory-sensitive operators.