A Tale of Two Scales: Reconciling Horizontal and Vertical Scaling for Inference Serving Systems

TL;DR

This paper tackles the challenge of sustain­ing low-latency DL inference in multi-model pipelines under highly variable workloads. It introduces Themis, a two-stage autoscaling system that first employs in-place vertical scaling to absorb bursts and then transitions to horizontal scaling to optimize resource usage, guided by offline latency profiles and a dynamic-programming–based optimizer. The IP formulation guarantees end-to-end latency SLOs while minimizing resource cost, with vertical and horizontal solvers executed based on workload status, and transitions driven by theoretical insights (Amdahl's law) and LSTM-based workload prediction. Real-world traces and a two-node testbed show Themis achieving more than 10x reduction in SLO violations and improved resource efficiency compared with state-of-the-art horizontal or vertical autoscalers, demonstrating a practical, pipeline-aware autoscaling solution for inference serving.

Abstract

Inference serving is of great importance in deploying machine learning models in real-world applications, ensuring efficient processing and quick responses to inference requests. However, managing resources in these systems poses significant challenges, particularly in maintaining performance under varying and unpredictable workloads. Two primary scaling strategies, horizontal and vertical scaling, offer different advantages and limitations. Horizontal scaling adds more instances to handle increased loads but can suffer from cold start issues and increased management complexity. Vertical scaling boosts the capacity of existing instances, allowing for quicker responses but is limited by hardware and model parallelization capabilities. This paper introduces Themis, a system designed to leverage the benefits of both horizontal and vertical scaling in inference serving systems. Themis employs a two-stage autoscaling strategy: initially using in-place vertical scaling to handle workload surges and then switching to horizontal scaling to optimize resource efficiency once the workload stabilizes. The system profiles the processing latency of deep learning models, calculates queuing delays, and employs different dynamic programming algorithms to solve the joint horizontal and vertical scaling problem optimally based on the workload situation. Extensive evaluations with real-world workload traces demonstrate over $10\times$ SLO violation reduction compared to the state-of-the-art horizontal or vertical autoscaling approaches while maintaining resource efficiency when the workload is stable.

Figures (11)

• Figure 1: Vertical scaling vs. horizontal scaling reaction time in case of workload bursts. Horizontal scaling is not responsive. The gray area indicates SLO violation.
• Figure 2: Using vertical and horizontal scaling jointly to absorb bursts and reduce operational costs. Vertical scaling provides responsiveness, while horizontal scaling provides cost efficiency. The gray area indicates SLO violation.
• Figure 3: An overview of the Themis architecture. The executor receives the requests and processes them. The monitor service collects the metric data from the stages. The optimizer makes both vertical and horizontal scaling decisions for the DL models. The Adapter enforces the scaling decisions by configuring the queues and the executor.
• Figure 4: Transition states between vertical and horizontal scaling strategies.
• Figure 5: LSTM inference result.