Harpagon: Minimizing DNN Serving Cost via Efficient Dispatching, Scheduling and Splitting

TL;DR

Harpagon tackles the problem of minimizing cloud serving cost for millisecond-latency DNN inference by introducing a three-level design: batch-aware request dispatching to minimize module latency, a module scheduler that supports multi-tuple configurations and residual optimizations (including a dummy generator and latency reassigner) to maximize throughput, and a latency-splitting mechanism based on latency-cost efficiency to allocate budgets across modules for multi-DNN applications. The system employs a throughput-cost dispatch policy that enables larger batch sizes and higher throughput under latency constraints, along with an optimization-driven approach to residual workloads and latency budgets. Empirical results show substantial cost reductions over state-of-the-art baselines and near-optimality compared with brute-force search on most workloads, with millisecond runtime, across heterogeneous GPU hardware. The work demonstrates practical impact for scalable, cost-efficient real-time DNN serving in cloud environments.

Abstract

Advances in deep neural networks (DNNs) have significantly contributed to the development of real-time video processing applications. Efficient scheduling of DNN workloads in cloud-hosted inference systems is crucial to minimizing serving costs while meeting application latency constraints. However, existing systems suffer from excessive module latency during request dispatching, low execution throughput during module scheduling, and wasted latency budget during latency splitting for multi-DNN application, which undermines their capability to minimize the serving cost. In this paper, we design a DNN inference system called Harpagon, which minimizes the serving cost under latency constraints with a three-level design. It first maximizes the batch collection rate with a batch-aware request dispatch policy to minimize the module latency. It then maximizes the module throughput with multi-tuple configurations and proper amount of dummy requests. It also carefully splits the end-to-end latency into per-module latency budget to minimize the total serving cost for multi-DNN applications. Evaluation shows that Harpagon outperforms the state of the art by 1.49 to 2.37 times in serving cost while satisfying the latency objectives. Additionally, compared to the optimal solution using brute force search, Harpagon derives the lower bound of serving cost for 91.5% workloads with millisecond level runtime.

Figures (12)

• Figure 1: Requests in the same batch have varying latency.
• Figure 2: (a) round-robin dispatch; (b) batch dispatch.
• Figure 3: Harpagon overview.
• Figure 4: Request dispatching results for throughput-cost dispatch policy (top) and round-robin dispatch policy (bottom).
• Figure 5: (a) The average normalized cost for Harpagon, four baseline systems and the optimal solution under 1131 workloads; (b) CDF of normalized cost.

Theorems & Definitions (4)

• Theorem 1
• proof
• Theorem 2
• proof