HybridServe: Efficient Serving of Large AI Models with Confidence-Based Cascade Routing

TL;DR

The paper addresses the prohibitive energy cost of serving giant DNNs in cloud environments. It introduces HybridServe, a confidence-based hybrid dataflow that routes inference requests across multiple model sizes (small to giant) and a dataflow planner that co-locates and replicates models to maximize throughput while respecting GPU constraints. Key contributions include a calibration-based confidence scoring framework that works without ground-truth labels, a threshold-search mechanism with Accuracy-Preserving and Energy-Optimization modes, and a polynomial-time planner for large-scale clusters; together they achieve up to 19.8x energy savings with accuracy comparable to giant-model serving. The work demonstrates practical impact by showing substantial energy and throughput improvements across ViT, T5, and GPT-style models on real hardware, enabling scalable, energy-efficient deployment of large AI services without additional fine-tuning.

Abstract

Giant Deep Neural Networks (DNNs), have become indispensable for accurate and robust support of large-scale cloud based AI services. However, serving giant DNNs is prohibitively expensive from an energy consumption viewpoint easily exceeding that of training, due to the enormous scale of GPU clusters needed to hold giant DNN model partitions and replicas. Existing approaches can either optimize energy efficiency or inference accuracy but not both. To overcome this status quo, we propose HybridServe, a novel hybrid DNN model serving system that leverages multiple sized versions (small to giant) of the model to be served in tandem. Through a confidence based hybrid model serving dataflow, HybridServe prefers to serve inference requests with energy-efficient smaller models so long as accuracy is not compromised, thereby reducing the number of replicas needed for giant DNNs. HybridServe also features a dataflow planner for efficient partitioning and replication of candidate models to maximize serving system throughput. Experimental results using a prototype implementation of HybridServe show that it reduces energy footprint by up to 19.8x compared to the state-of-the-art DNN model serving systems while matching the accuracy of serving solely with giant DNNs.

Figures (8)

• Figure 1: Schematic of a typical giant DNN serving system.
• Figure 2: HybridServe system design overview.
• Figure 3: Overview of hybrid model serving dataflow.
• Figure 4: Latency performance
• Figure 5: Throughput performance