Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NAS-BNN: Neural Architecture Search for Binary Neural Networks

Zhihao Lin, Yongtao Wang, Jinhe Zhang, Xiaojie Chu, Haibin Ling

TL;DR

NAS-BNN introduces a binary-specific NAS framework that designs a binary-friendly search space and employs three training strategies—Bi-Teacher, Bi-Transformation, and Weight Normalization—plus non-decreasing channel constraints and two deployment pipelines. The approach yields binary models that outperform prior BNNs on ImageNet and MS COCO, with top-1 accuracies approaching or exceeding 70% at moderate OPs (e.g., $70.80\%$ at $180$M OPs) and MS COCO mAP scores above $32\%$, while avoiding reliance on full-precision teachers. The method achieves these gains with a total search cost of around $240$ GPU days and demonstrates strong transferability to object detection. Overall, NAS-BNN provides a practical, scalable path to high-performance BNNs suitable for resource-constrained deployment.

Abstract

Binary Neural Networks (BNNs) have gained extensive attention for their superior inferencing efficiency and compression ratio compared to traditional full-precision networks. However, due to the unique characteristics of BNNs, designing a powerful binary architecture is challenging and often requires significant manpower. A promising solution is to utilize Neural Architecture Search (NAS) to assist in designing BNNs, but current NAS methods for BNNs are relatively straightforward and leave a performance gap between the searched models and manually designed ones. To address this gap, we propose a novel neural architecture search scheme for binary neural networks, named NAS-BNN. We first carefully design a search space based on the unique characteristics of BNNs. Then, we present three training strategies, which significantly enhance the training of supernet and boost the performance of all subnets. Our discovered binary model family outperforms previous BNNs for a wide range of operations (OPs) from 20M to 200M. For instance, we achieve 68.20% top-1 accuracy on ImageNet with only 57M OPs. In addition, we validate the transferability of these searched BNNs on the object detection task, and our binary detectors with the searched BNNs achieve a novel state-of-the-art result, e.g., 31.6% mAP with 370M OPs, on MS COCO dataset. The source code and models will be released at https://github.com/VDIGPKU/NAS-BNN.

NAS-BNN: Neural Architecture Search for Binary Neural Networks

TL;DR

NAS-BNN introduces a binary-specific NAS framework that designs a binary-friendly search space and employs three training strategies—Bi-Teacher, Bi-Transformation, and Weight Normalization—plus non-decreasing channel constraints and two deployment pipelines. The approach yields binary models that outperform prior BNNs on ImageNet and MS COCO, with top-1 accuracies approaching or exceeding 70% at moderate OPs (e.g., at M OPs) and MS COCO mAP scores above , while avoiding reliance on full-precision teachers. The method achieves these gains with a total search cost of around GPU days and demonstrates strong transferability to object detection. Overall, NAS-BNN provides a practical, scalable path to high-performance BNNs suitable for resource-constrained deployment.

Abstract

Binary Neural Networks (BNNs) have gained extensive attention for their superior inferencing efficiency and compression ratio compared to traditional full-precision networks. However, due to the unique characteristics of BNNs, designing a powerful binary architecture is challenging and often requires significant manpower. A promising solution is to utilize Neural Architecture Search (NAS) to assist in designing BNNs, but current NAS methods for BNNs are relatively straightforward and leave a performance gap between the searched models and manually designed ones. To address this gap, we propose a novel neural architecture search scheme for binary neural networks, named NAS-BNN. We first carefully design a search space based on the unique characteristics of BNNs. Then, we present three training strategies, which significantly enhance the training of supernet and boost the performance of all subnets. Our discovered binary model family outperforms previous BNNs for a wide range of operations (OPs) from 20M to 200M. For instance, we achieve 68.20% top-1 accuracy on ImageNet with only 57M OPs. In addition, we validate the transferability of these searched BNNs on the object detection task, and our binary detectors with the searched BNNs achieve a novel state-of-the-art result, e.g., 31.6% mAP with 370M OPs, on MS COCO dataset. The source code and models will be released at https://github.com/VDIGPKU/NAS-BNN.
Paper Structure (28 sections, 6 equations, 3 figures, 6 tables)

This paper contains 28 sections, 6 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Binary neural network
  4. Neural architecture search
  5. Methodology
  6. Problem definition
  7. Search space
  8. Training for binary supernet
  9. Bi-Teacher for sandwich rule
  10. Bi-Transformation for shared weights
  11. Weight Normalization
  12. Deployment pipelines
  13. Experiments
  14. Experimental setups
  15. Dataset
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Comparison of the cases with different channel widths in a Bi-Real module. "$\times$" denotes the binary convolution, "C" denotes the tiling channel concatenation, and "$+$" denotes the element-wise addition. (a) and (b) illustrate equal and increasing cases, respectively, while (c) illustrates the case where channel width decreases. It shows a significant loss of feature information in (c) (i.e., the portion denoted by the slash).
  • Figure 2: Illustration of the process to obtain full-precision and binary weights from a shared weight.
  • Figure 3: Impact of the proposed training techniques on ImageNet-100. (a-c) We randomly sample 1000 subnets and directly evaluate the validation accuracy with inherited parameters from the supernet. (d) The training accuracy of random subnets with different teachers.