End-to-end fully-binarized network design: from Generic Learned Thermometer to Block Pruning

TL;DR

The paper tackles the input-data bottleneck in Binary Neural Networks by introducing Generic Learned Thermometer (GLT), a trainable nonlinear input encoding that learns thresholds $t_i$ from latent parameters via $\bar{t} = \tilde{t} / \sum \tilde{t}$ and $t_i = \sum_{j=1}^i \bar{t}_j$, enabling global tone mapping prior to 1-bit computations. It pairs GLT with a block-pruning strategy that replaces complex blocks with lightweight grouped convolutions, guided by a distributional (KL) distillation loss $\mathcal{L}_{distr}$ together with cross-entropy $\mathcal{L}_{ce}$ in the total loss $\mathcal{L} = (1-\lambda)\mathcal{L}_{ce} + \lambda\mathcal{L}_{distr}$, operating in stages across blocks with temperature $T=8$ and $\lambda=0.5$. Empirically, GLT improves accuracy on VWW and STL-10, especially for gamma-inversed inputs where gains are larger, and the block-pruning pipeline yields highly compact, fully-binarized models (sub-1MB) with modest accuracy loss (e.g., ~73% after aggressive pruning). The approach demonstrates end-to-end viability for in-sensor, always-on inference, providing nonlinear input encoding and hardware-friendly pruning that together deliver high efficiency without severe performance degradation.

Abstract

Existing works on Binary Neural Network (BNN) mainly focus on model's weights and activations while discarding considerations on the input raw data. This article introduces Generic Learned Thermometer (GLT), an encoding technique to improve input data representation for BNN, relying on learning non linear quantization thresholds. This technique consists in multiple data binarizations which can advantageously replace a conventional Analog to Digital Conversion (ADC) that uses natural binary coding. Additionally, we jointly propose a compact topology with light-weight grouped convolutions being trained thanks to block pruning and Knowledge Distillation (KD), aiming at reducing furthermore the model size so as its computational complexity. We show that GLT brings versatility to the BNN by intrinsically performing global tone mapping, enabling significant accuracy gains in practice (demonstrated by simulations on the STL-10 and VWW datasets). Moreover, when combining GLT with our proposed block-pruning technique, we successfully achieve lightweight (under 1Mb), fully-binarized models with limited accuracy degradation while being suitable for in-sensor always-on inference use cases.

Figures (5)

• Figure 1: Three examples of encoding techniques for input image data binary representation. Here, for the sake of simplicity, input pixel dynamic range is considered between 0 and 255.
• Figure 2: Nonlinear ramp ADC top-level view schematic.
• Figure 3: Block pruning with an auxiliary lightweight grouped convolution (LWC) module. Note that our pruning method can be applied to an arbitrary network.
• Figure 4: Encoding curves of MUXORNet-11 GLT layers trained on gamma-inversed STL-10. Black: fixed linear curve FracBNN, Red/Green/Blue: learned curves of R/G/B channels.
• Figure 5: Trade-off curve for model size/BOPs reduction and accuracy loss of pruned MUXORNet-11 (original acc: 78.5$\%$).