Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WVA: A Global Optimization Control Plane for llmd

Abhishek Malvankar, Lionel Villard, Mohammed Abdi, Evgeny Shindin, Braulio Dumba, Vishakha Ramani, Asser Tantawi, Tamar Eilam

TL;DR

The Workload Variant Autoscaler (WVA) is introduced, a specialized control plane co-designed withllmd that tightly couples scaling decisions with the inference server's internal saturation state and intrinsically reduces overall power consumption by prioritizing lower-cost, energy-efficient hardware variants over homogeneous scaling on high-end accelerators.

Abstract

As Large Language Models (LLMs) scale to handle massive concurrent traffic, optimizing the infrastructure required for inference has become a primary challenge. To manage the high cost of GPU resources while ensuring strict service-level objectives (SLOs), operators increasingly deploy models across heterogeneous hardware clusters that multiplex latency-sensitive online requests and throughput-oriented offline requests. However, traditional resource-centric autoscalers like the Kubernetes horizontal pod autoscaler (HPA) do not consider application-specific SLOs, hardware heterogeneity, or internal engine state (like KV cache utilization) globally. This leads to unnecessary scaling, severe resource underutilization, and disrupted stateful inference. To address these limitations, we introduce the Workload Variant Autoscaler (WVA), a specialized control plane co-designed with \texttt{llmd} that tightly couples scaling decisions with the inference server's internal saturation state. By utilizing proactive headroom-based scaling and fragmentation-aware scale-down, our experiments demonstrate that WVA achieves a \textbf{37\% improvement in effective throughput} and a \textbf{10x reduction in request failures} compared to HPA. Furthermore, WVA's cost-aware tiering intrinsically reduces overall power consumption by prioritizing lower-cost, energy-efficient hardware variants over homogeneous scaling on high-end accelerators.

WVA: A Global Optimization Control Plane for llmd

TL;DR

The Workload Variant Autoscaler (WVA) is introduced, a specialized control plane co-designed withllmd that tightly couples scaling decisions with the inference server's internal saturation state and intrinsically reduces overall power consumption by prioritizing lower-cost, energy-efficient hardware variants over homogeneous scaling on high-end accelerators.

Abstract

As Large Language Models (LLMs) scale to handle massive concurrent traffic, optimizing the infrastructure required for inference has become a primary challenge. To manage the high cost of GPU resources while ensuring strict service-level objectives (SLOs), operators increasingly deploy models across heterogeneous hardware clusters that multiplex latency-sensitive online requests and throughput-oriented offline requests. However, traditional resource-centric autoscalers like the Kubernetes horizontal pod autoscaler (HPA) do not consider application-specific SLOs, hardware heterogeneity, or internal engine state (like KV cache utilization) globally. This leads to unnecessary scaling, severe resource underutilization, and disrupted stateful inference. To address these limitations, we introduce the Workload Variant Autoscaler (WVA), a specialized control plane co-designed with \texttt{llmd} that tightly couples scaling decisions with the inference server's internal saturation state. By utilizing proactive headroom-based scaling and fragmentation-aware scale-down, our experiments demonstrate that WVA achieves a \textbf{37\% improvement in effective throughput} and a \textbf{10x reduction in request failures} compared to HPA. Furthermore, WVA's cost-aware tiering intrinsically reduces overall power consumption by prioritizing lower-cost, energy-efficient hardware variants over homogeneous scaling on high-end accelerators.
Paper Structure (38 sections, 4 equations, 6 figures)

This paper contains 38 sections, 4 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Background: LLM Inference Dynamics
  4. Motivation: Balancing Cost and SLOs
  5. State of the Art: The Shortcomings of Legacy Scaling
  6. System Model
  7. The Variant Abstraction
  8. Saturation Model (Default Strategy)
  9. Global Optimization Framework
  10. Headroom-Based Capacity Management
  11. Fragmentation-Aware Safety
  12. End-to-End Workflow
  13. Configuration Interface
  14. Architecture
  15. Architectural Overview
  16. ...and 23 more sections

Figures (6)

  • Figure 1: WVA integrated with llmd
  • Figure 2: Saturation Reactivity: WVA response time to an unpredicted, near-instantaneous onset of high-concurrency requests.
  • Figure 3: Cost-Aware Scaling: A100s scale up before H100s to minimize inference cost under ramp-up load.
  • Figure 4: Throughput Stability: WVA (Red) maintains higher, more consistent throughput compared to HPA (Orange).
  • Figure 5: Request Stability: WVA (Red) minimizes drops via saturation buffers and safe termination, while HPA (Orange) incurs failures due to throttling and instability.
  • ...and 1 more figures