SC2 Benchmark: Supervised Compression for Split Computing

TL;DR

The paper addresses the challenge of running deep learning inference on resource-constrained devices by formalizing supervised compression for split computing (SC2) and proposing evaluation criteria that jointly optimize encoder size, transmitted data, and task accuracy. It introduces a comprehensive benchmark across 10 methods and 3 computer vision tasks, revealing that Entropic Student consistently delivers strong supervised rate-distortion performance while maintaining a small encoder footprint; hyperprior enhancements further improve compression at encoder cost. A three-way ExR-D tradeoff is proposed to visualize and select bottleneck placements and architectures that favor edge-side computation. The authors release sc2bench to standardize SC2 experimentation and facilitate reproducible research, underscoring the practical significance for efficient mobile-edge inference in real-world systems.

Abstract

With the increasing demand for deep learning models on mobile devices, splitting neural network computation between the device and a more powerful edge server has become an attractive solution. However, existing split computing approaches often underperform compared to a naive baseline of remote computation on compressed data. Recent studies propose learning compressed representations that contain more relevant information for supervised downstream tasks, showing improved tradeoffs between compressed data size and supervised performance. However, existing evaluation metrics only provide an incomplete picture of split computing. This study introduces supervised compression for split computing (SC2) and proposes new evaluation criteria: minimizing computation on the mobile device, minimizing transmitted data size, and maximizing model accuracy. We conduct a comprehensive benchmark study using 10 baseline methods, three computer vision tasks, and over 180 trained models, and discuss various aspects of SC2. We also release sc2bench, a Python package for future research on SC2. Our proposed metrics and package will help researchers better understand the tradeoffs of supervised compression in split computing.

Figures (7)

• Figure 1: Input compression (top) vs. SC2: supervised compression for split computing (bottom) for image classification. The input compression approach reconstructs the input image on the could/edge server whereas the SC2 approach produces an compressible feature representation suitable for the supervised task. Note that the training process is done offline, and then the model will be split for deployment.
• Figure 2: Bitrate comparison between a supervised compression (SC) model matsubara2022supervised and an input compression (IC) model balle2018variational. We plot the difference of the bits allocated for each pixel, exemplified on three images. Areas where the SC model allocates fewer and more bits for the given image are indicated in blue and red, respectively (best viewed in PDF). It is apparent how supervised compression allocates more bits to information relevant to the supervised object recognition goal.
• Figure 3: SC2 for image classification on ImageNet (ILSVRC 2012). We show the supervised R-D tradeoff (left), the ExR-D tradeoff (middle), and the full three-way tradeoff (right). In all cases, we used ResNet-50 as our reference model. Grey lines denote projections. Entropic Student performed best in R-D and ExR-D performance.
• Figure 4: SC2 for object detection on COCO 2017. We show the supervised R-D tradeoff (left), the ExR-D tradeoff (middle), and the full three-way tradeoff (right). In all cases, we used Faster R-CNN with ResNet-50 and FPN as our reference model. Grey lines denote projections. Entropic Student performed best in R-D and ExR-D performance.
• Figure 6: Entropic Student vs. ResNet-50 with entropy bottleneck (EB) introduced to different layers. Simply introducing EBs to its late layers e.g., layer4 and avgpool improved the conventional R-D tradeoff (left), which results in most of the layers in the model to be deployed on weak local devices. Our proposed ExR-D and three-way tradeoffs penalize such configurations (middle and right).