BD-Net: Has Depth-Wise Convolution Ever Been Applied in Binary Neural Networks?

TL;DR

This work tackles the challenge of binarizing depth-wise convolutions in Binary Neural Networks (BNNs), a bottleneck for truly lightweight models. It introduces BD-Net, combining a pre-BN residual with a 1.58-bit dual-binary convolution to stabilize training and boost expressive capacity without extra parameters. BD-Net achieves remarkable efficiency, delivering about 33M OPs on ImageNet with MobileNet V1 and outperforming prior 1-bit methods by up to several percentage points across multiple datasets. The approach also extends to other efficient architectures via broadcast residuals, indicating broad applicability for edge-friendly binary CNNs.

Abstract

Recent advances in model compression have highlighted the potential of low-bit precision techniques, with Binary Neural Networks (BNNs) attracting attention for their extreme efficiency. However, extreme quantization in BNNs limits representational capacity and destabilizes training, posing significant challenges for lightweight architectures with depth-wise convolutions. To address this, we propose a 1.58-bit convolution to enhance expressiveness and a pre-BN residual connection to stabilize optimization by improving the Hessian condition number. These innovations enable, to the best of our knowledge, the first successful binarization of depth-wise convolutions in BNNs. Our method achieves 33M OPs on ImageNet with MobileNet V1, establishing a new state-of-the-art in BNNs by outperforming prior methods with comparable OPs. Moreover, it consistently outperforms existing methods across various datasets, including CIFAR-10, CIFAR-100, STL-10, Tiny ImageNet, and Oxford Flowers 102, with accuracy improvements of up to 9.3 percentage points.

Figures (9)

• Figure 1: Top-1 accuracy vs. operations (millions of OPs) across various datasets. Colors denote datasets: CIFAR-10 (blue), CIFAR-100 (orange), STL-10 (green), Tiny ImageNet (red), and Flowers-102 (purple). Marker shapes represent models: BD-Net (stars), ReActNet (squares), and AdaBin (diamonds). A blue star, for example, indicates BD-Net applied to CIFAR-10. BD-Net consistently achieves superior accuracy with significantly fewer operations across all datasets.
• Figure 2: Loss landscapes of (a) binarized MobileNet V1 and (b) full-precision MobileNet V1.
• Figure 3: Architecture comparison of binary depth-wise convolution layers.
• Figure 4: Image visualization on ImageNet validation dataset. (a) Converted to grayscale, (b) Binarized using Otsu's algorithm, (c) Displayed using an appropriate threshold to represent in 1.58-bit.
• Figure 5: Graph of the top-10 hessian eigenvalues for several networks.