Leveraging Stochastic Depth Training for Adaptive Inference

TL;DR

The paper tackles the challenge of adaptive, per-input inference at the edge by exploiting models trained with Stochastic Depth to become resilient to arbitrary layer skipping. It replaces costly, learnable gating or decision networks with a zero-overhead gating mechanism and a design-time Pareto-front exploration of skipping configurations, enabling a single-model, time-predictable inference. At runtime, a lightweight algorithm selects among Pareto configurations based on current edge conditions, achieving significant improvements in energy efficiency and throughput (up to 2× power savings and nearly 2× more inferences in some cases) while maintaining acceptable accuracy. This approach offers predictable, controllable adaptive inference suitable for resource-constrained edge deployments and demonstrates concrete gains on CIFAR-10/100 with ResNets, compiled via IREE for edge devices. The work bridges stochastic-depth training with practical, zero-overhead adaptive inference, enabling deployment of large models with tunable accuracy-time-energy profiles.

Abstract

Dynamic DNN optimization techniques such as layer-skipping offer increased adaptability and efficiency gains but can lead to i) a larger memory footprint as in decision gates, ii) increased training complexity (e.g., with non-differentiable operations), and iii) less control over performance-quality trade-offs due to its inherent input-dependent execution. To approach these issues, we propose a simpler yet effective alternative for adaptive inference with a zero-overhead, single-model, and time-predictable inference. Central to our approach is the observation that models trained with Stochastic Depth -- a method for faster training of residual networks -- become more resilient to arbitrary layer-skipping at inference time. We propose a method to first select near Pareto-optimal skipping configurations from a stochastically-trained model to adapt the inference at runtime later. Compared to original ResNets, our method shows improvements of up to 2X in power efficiency at accuracy drops as low as 0.71%.

Figures (6)

• Figure 1: Accuracy curve for skipping blocks in ResNet-110 (on CIFAR-10) trained with traditional and stochastic depth procedures (shaded area give the distribution for 500 skipping configurations).
• Figure 2: Traditional skipping (adapted from skipnet). Batch-normalization and activation layers omitted for simplicity.
• Figure 3: Overview of the proposed framework.
• Figure 4: A toy example of an adaptive ResNet with two skippable blocks.
• Figure 5: Our proposed skipping. Batch-normalization and activation layers omitted for simplicity.