Accelerating End-Cloud Collaborative Inference via Near Bubble-free Pipeline Optimization

TL;DR

COACH tackles the bottleneck of pipeline bubbles in end‑cloud collaborative DNN inference by coupling an offline recursive divide‑and‑conquer partitioning/quantization optimization with an online context‑aware quantization adjustment and caching strategy. It minimizes bubble metrics $B_c(V_p)$ and $B_t(V_p)$ while constraining latency to $\max\{T_e, T_t, T_c\}$, achieving $O(cn)$ offline complexity and enabling bubble‑free scheduling in DAG topologies. The online component uses label semantic centers $\mathbf{T_c}$, task similarity $t_j$, and task separability $S$ to adapt quantization in real time, supported by an early‑exit policy to further reduce data transmissions. Empirically, COACH delivers up to 1.7x–2.1x faster inference and 2.1x–2.5x throughput improvements over baselines across dynamic networks and bandwidth conditions.

Abstract

End-cloud collaboration offers a promising strategy to enhance the Quality of Service (QoS) in DNN inference by offloading portions of the inference workload from end devices to cloud servers. Despite the potential, the complex model architectures and dynamic network conditions will introduce numerous bubbles (\ie, idle waiting time) in pipeline execution, resulting in inefficient resource utilization and degraded QoS. To address these challenges, we introduce a novel framework named COACH, designed for near bubble-free pipeline collaborative inference, thereby achieving low inference latency and high system throughput. Initially, COACH employs an \textit{offline} component that utilizes an efficient recursive divide-and-conquer algorithm to optimize both model partitioning and transmission quantization, aiming to minimize the occurrence of pipeline bubbles. Subsequently, the \textit{online} component in COACH employs an adaptive quantization adjustment and a context-aware caching strategy to further stabilize pipeline execution. Specifically, COACH analyzes the correlation between intermediate data and label semantic centers in the cache, along with its influence on the quantization adjustment, thereby effectively accommodating network fluctuations. Our experiments demonstrate the efficacy of COACH in reducing inference latency and enhancing system throughput. Notably, while maintaining comparable accuracy, COACH achieves up to 1.7x faster inference and 2.1x higher system throughput than baselines.

Figures (9)

• Figure 1: Data correlation visualization on the UCF101 dataset with the ResNet101 model.
• Figure 2: Three-stage collaborative inference processes with pipeline scheduling.
• Figure 3: Overview and inference workflow of COACH.
• Figure 4: Illustrating DNN partitioning with virtual blocks and layer parallel execution.
• Figure 5: Adaptability of COACH and baselines in dynamic network conditions.