Where to Split? A Pareto-Front Analysis of DNN Partitioning for Edge Inference

TL;DR

This work addresses the challenge of deploying DNNs on resource-constrained edge devices by reframing model partitioning as a multi-objective optimization problem over latency and throughput. It introduces ParetoPipe, an open-source framework that maps the entire latency-throughput Pareto frontier using pipeline partitioning across heterogeneous devices and dual backends (PyTorch RPC and a lightweight socket implementation). Through extensive experiments on Raspberry Pi clusters and an edge GPU server, the authors reveal how network variability shifts optimal partition points and demonstrate substantial gains when using a network-aware approach; they also show that custom backends can drastically reduce coordination overhead. The study provides practical insights and a foundation for robust, real-world edge inference, including guidance on partitioning strategies under adverse network conditions and a path toward adaptive, energy-aware multi-device deployments.

Abstract

The deployment of deep neural networks (DNNs) on resource-constrained edge devices is frequently hindered by their significant computational and memory requirements. While partitioning and distributing a DNN across multiple devices is a well-established strategy to mitigate this challenge, prior research has largely focused on single-objective optimization, such as minimizing latency or maximizing throughput. This paper challenges that view by reframing DNN partitioning as a multi-objective optimization problem. We argue that in real-world scenarios, a complex trade-off between latency and throughput exists, which is further complicated by network variability. To address this, we introduce ParetoPipe, an open-source framework that leverages Pareto front analysis to systematically identify optimal partitioning strategies that balance these competing objectives. Our contributions are threefold: we benchmark pipeline partitioned inference on a heterogeneous testbed of Raspberry Pis and a GPU-equipped edge server; we identify Pareto-optimal points to analyze the latency-throughput trade-off under varying network conditions; and we release a flexible, open-source framework to facilitate distributed inference and benchmarking. This toolchain features dual communication backends, PyTorch RPC and a custom lightweight implementation, to minimize overhead and support broad experimentation.

Figures (7)

• Figure 1: The ParetoPipe system architecture. The orchestrator partitions a DNN model across a set of heterogeneous worker nodes, creating a multi-stage pipeline. Each worker processes its assigned stage and forwards intermediate results to the next device in the pipeline.
• Figure 2: Block-wise profiling results of DNN models on Raspberry Pi (CPU) and edge server (GPU).
• Figure 3: Pareto Frontiers for DNN Partitioning on a Pi-to-Pi configuration.
• Figure 4: Pareto Frontiers for DNN Partitioning on a Pi-to-GPU configuration.
• Figure 5: Pi-to-Pi Pareto Frontier under simulated network duress (200ms Latency, 5 Mbit/s Bandwidth).