Slimmable Neural Networks

TL;DR

This work addresses runtime variability across devices by enabling a single neural network to operate at multiple channel widths. It introduces Slimmable Neural Networks with switchable batch normalization to align statistics across configurations, enabling joint training. Empirically, the approach matches or surpasses independently trained models on ImageNet and extends effectively to COCO detection and segmentation tasks, demonstrating strong cross-domain applicability. The method provides on-device, latency-aware trade-offs with minimal overhead and broad potential for extension to related areas like pruning and unsupervised learning.

Abstract

We present a simple and general method to train a single neural network executable at different widths (number of channels in a layer), permitting instant and adaptive accuracy-efficiency trade-offs at runtime. Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization. At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models. Our trained networks, named slimmable neural networks, achieve similar (and in many cases better) ImageNet classification accuracy than individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths respectively. We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters. Lastly we visualize and discuss the learned features of slimmable networks. Code and models are available at: https://github.com/JiahuiYu/slimmable_networks

Figures (4)

• Figure 1: Illustration of slimmable neural networks. The same model can run at different widths (number of active channels), permitting instant and adaptive accuracy-efficiency trade-offs.
• Figure 2: Training and validation curves of slimmable networks. Left shows the training error of the largest switch. Right shows testing errors on validation set with different switches. For naive approach, the training is stable (left) but testing error is high (right, zoomed). Slimmable networks trained with S-BN have stable and rank-preserved testing accuracy across all training iterations.
• Figure 3: Top-activated images for same channel $3\_9$ in different switches in S-MobileNet v1. Different rows represent results from different switches. Images with red outlines are mis-classified. Note that the white color in RGB is $[255,255,255]$, yellow in RGB is $[255,255,0]$.
• Figure 4: Values of BN parameters in different switches. We show BN values of both shallow (left, BN $1\_1$ to $1\_8$) and deep (right, BN $12\_1$ to $12\_8$) layers of S-MobileNet v1.