Loki: A System for Serving ML Inference Pipelines with Hardware and Accuracy Scaling

TL;DR

Loki tackles the practical challenge of serving ML inference pipelines under fluctuating demand by unifying hardware scaling with accuracy scaling within a pipeline-aware framework. It introduces a MILP-based Resource Manager that jointly optimizes model variants, replication, and batch sizes, and a MostAccurateFirst routing scheme with opportunistic rerouting to maximize end-to-end pipeline accuracy and minimize SLO violations. Empirical results show Loki can boost effective capacity by up to $2.7\times$ and cut SLO violations by up to $10\times$ compared to pipeline-agnostic baselines, while reducing active servers in off-peak periods by up to $2.67\times$, with modest accuracy trade-offs. The work demonstrates the practical value of integrating resource-aware accuracy scaling with hardware scaling for complex inference pipelines in real-time settings.

Abstract

The rapid adoption of machine learning (ML) has underscored the importance of serving ML models with high throughput and resource efficiency. Traditional approaches to managing increasing query demands have predominantly focused on hardware scaling, which involves increasing server count or computing power. However, this strategy can often be impractical due to limitations in the available budget or compute resources. As an alternative, accuracy scaling offers a promising solution by adjusting the accuracy of ML models to accommodate fluctuating query demands. Yet, existing accuracy scaling techniques target independent ML models and tend to underperform while managing inference pipelines. Furthermore, they lack integration with hardware scaling, leading to potential resource inefficiencies during low-demand periods. To address the limitations, this paper introduces Loki, a system designed for serving inference pipelines effectively with both hardware and accuracy scaling. Loki incorporates an innovative theoretical framework for optimal resource allocation and an effective query routing algorithm, aimed at improving system accuracy and minimizing latency deadline violations. Our empirical evaluation demonstrates that through accuracy scaling, the effective capacity of a fixed-size cluster can be enhanced by more than $2.7\times$ compared to relying solely on hardware scaling. When compared with state-of-the-art inference-serving systems, Loki achieves up to a $10\times$ reduction in Service Level Objective (SLO) violations, with minimal compromises on accuracy and while fulfilling throughput demands.

Figures (8)

• Figure 1: For a traffic analysis pipeline that consists of two sequential tasks, Loki first accomodates the increase in query demand by using hardware scaling. When demand increases further, Loki successively decreases the accuracy of each task of the pipeline to increase throughput to meet demand. Phase 2 decreases the $2^{\text{nd}}$ task's accuracy as it causes smaller end-to-end accuracy drop.
• Figure 2: Examples of inference pipelines
• Figure 3: Accuracy-throughput tradeoff for EfficientNet model variants as profiled on an NVIDIA V100 GPU
• Figure 4: System Architecture of Loki
• Figure 5: End-to-end comparison on the traffic analysis pipeline. Dotted vertical lines show transition between hardware and accuracy scaling. Loki achieves an increase of $2.5\times$ in effective capacity compared to InferLine that performs hardware scaling alone and reduces SLO violations by up to $10\times$ compared to Proteus that performs pipeline-unaware accuracy scaling.