Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving

TL;DR

Apparate addresses the latency-throughput tension in ML inference by automatically injecting and managing early exits (EEs) to reduce per-request latency without sacrificing throughput or accuracy. It relies on continual, per-input feedback from EE usage to drive two decoupled runtime adaptation loops: fast threshold tuning (accuracy-focused) and periodic ramp adjustments (latency-focused), enabling broad applicability across CV, NLP classification, and generative workloads. The system supports diverse architectures by placing lightweight ramps at cut-vertex locations and reusing final prediction layers, while maintaining compatibility with batching and other efficiency techniques. Empirical results show substantial median latency reductions (up to ~90% in CV, ~24% in NLP, and ~78% per-token improvements in generative tasks) with minimal throughput impact and strict accuracy constraints; Apparate also demonstrates competitive performance against or superiority to existing EE strategies and releases open-source tooling for broader adoption.

Abstract

Machine learning (ML) inference platforms are tasked with balancing two competing goals: ensuring high throughput given many requests, and delivering low-latency responses to support interactive applications. Unfortunately, existing platform knobs (e.g., batch sizes) fail to ease this fundamental tension, and instead only enable users to harshly trade off one property for the other. This paper explores an alternate strategy to taming throughput-latency tradeoffs by changing the granularity at which inference is performed. We present Apparate, a system that automatically applies and manages early exits (EEs) in ML models, whereby certain inputs can exit with results at intermediate layers. To cope with the time-varying overhead and accuracy challenges that EEs bring, Apparate repurposes exits to provide continual feedback that powers several novel runtime monitoring and adaptation strategies. Apparate lowers median response latencies by 40.5--91.5% and 10.0--24.2% for diverse CV and NLP classification workloads, and median time-per-token latencies by 22.6--77.9% for generative scenarios, without affecting throughputs or violating tight accuracy constraints.

Figures (19)

• Figure 1: Throughput-latency tradeoff in model serving. Results show serving times with batch sizes of 1--16.
• Figure 2: Tuning platform knobs lowers latencies but harms throughput. Results vary TF-Serve's $max\_batch\_size$ from 4--16. Gray lines show min serving time per model (batch=1). CV uses a random corpus video; NLP uses Amazon reviews amazon_reviews.
• Figure 3: Early exits enable termination of inputs at intermediate layers, lowering both compute and latency.
• Figure 4: EEs can lower latencies without harming throughput. Results modulate latencies from TF-Serve with original/vanilla models (Figure \ref{['fig:tuning']}) based on optimal exiting.
• Figure 5: Optimal EE configurations change frequently. Workloads use 64-request chunks. Dot presence shows a ramp that was part of the optimal config for a chunk, while transparencies indicate threshold values (opaque is higher).