Layerwise Progressive Freezing Enables STE-Free Training of Deep Binary Neural Networks

TL;DR

The paper tackles the challenge of training deep binary neural networks without straight-through estimators by introducing StoMPP, a layerwise progressive freezing method with stochastic masking. StoMPP preserves learning signals by gradually hardening weights and activations while backpropagating through unfrozen proxies, thereby avoiding STE-induced forward/backward mismatch and gradient blockades. Empirically, StoMPP delivers substantial depth-aware improvements over STE, especially for full BNNs, and demonstrates compatibility with architectures like Bi-Real Net, though interactions with some STE refinements can be non-additive. The approach offers an estimator-free, scalable pathway to efficient binary networks with robust optimization dynamics and practical training overhead. This has practical impact for energy-efficient inference on deep networks, enabling STE-free training of binary models without sacrificing performance as models scale in depth.

Abstract

We investigate progressive freezing as an alternative to straight-through estimators (STE) for training binary networks from scratch. Under controlled training conditions, we find that while global progressive freezing works for binary-weight networks, it fails for full binary neural networks due to activation-induced gradient blockades. We introduce StoMPP (Stochastic Masked Partial Progressive Binarization), which uses layerwise stochastic masking to progressively replace differentiable clipped weights/activations with hard binary step functions, while only backpropagating through the unfrozen (clipped) subset (i.e., no straight-through estimator). Under a matched minimal training recipe, StoMPP improves accuracy over a BinaryConnect-style STE baseline, with gains that increase with depth (e.g., for ResNet-50 BNN: +18.0 on CIFAR-10, +13.5 on CIFAR-100, and +3.8 on ImageNet; for ResNet-18: +3.1, +4.7, and +1.3). For binary-weight networks, StoMPP achieves 91.2\% accuracy on CIFAR-10 and 69.5\% on CIFAR-100 with ResNet-50. We analyze training dynamics under progressive freezing, revealing non-monotonic convergence and improved depth scaling under binarization constraints.

Figures (14)

• Figure 1: Comparison of masking strategies in StoMPP. Top: Global stochastic masking randomly freezes activations (nodes) and weights (edges) throughout training. Bottom: Layer-wise stochastic masking freezes entire layers sequentially from input to output. Blue indicates active gradient paths, red indicates frozen elements (sign function), and orange indicates edges blocked by frozen target nodes.
• Figure 2: (a) Hyperparameter sweep for BNNs under the global mask on CIFAR-100 with ResNet18. We vary the freezing schedule $p(t)$ and refresh rate $r$ for StoMPP and report Top-1 test accuracy (%). (b--d) Accuracy trajectories of CIFAR-100 on ResNets, trained with STE and StoMPP under the same training recipe. StoMPP exhibits a sawtooth pattern corresponding to progressive freezing, while STE improves more smoothly over training. The dashed line represents the fully quantized StoMPP network accuracy.
• Figure 3: Sweep of epochs under training scheme; solid lines show train accuracy and dashed lines show test accuracy. Note 50 epochs is not included for StoMPP ResNet50 as there are not enough epochs to apply at least one epoch per layer of layerwise masking.
• Figure 4: Testing Accuracy Curves for BWNs
• Figure 5: Training Accuracy Curves for BWNs