Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CycleBNN: Cyclic Precision Training in Binary Neural Networks

Federico Fontana, Romeo Lanzino, Anxhelo Diko, Gian Luca Foresti, Luigi Cinque

TL;DR

The CycleBNN is introduced, an innovative methodology integrating BNNs with cyclic precision training, designed to enhance training efficiency while minimizing the loss in performance, and a path towards faster, more accessible training of efficient networks, accelerating the development of practical applications.

Abstract

This paper works on Binary Neural Networks (BNNs), a promising avenue for efficient deep learning, offering significant reductions in computational overhead and memory footprint to full precision networks. However, the challenge of energy-intensive training and the drop in performance have been persistent issues. Tackling the challenge, prior works focus primarily on task-related inference optimization. Unlike prior works, this study offers an innovative methodology integrating BNNs with cyclic precision training, introducing the CycleBNN. This approach is designed to enhance training efficiency while minimizing the loss in performance. By dynamically adjusting precision in cycles, we achieve a convenient trade-off between training efficiency and model performance. This emphasizes the potential of our method in energy-constrained training scenarios, where data is collected onboard and paves the way for sustainable and efficient deep learning architectures. To gather insights on CycleBNN's efficiency, we conduct experiments on ImageNet, CIFAR-10, and PASCAL-VOC, obtaining competitive performances while using 96.09\% less operations during training on ImageNet, 88.88\% on CIFAR-10 and 96.09\% on PASCAL-VOC. Finally, CycleBNN offers a path towards faster, more accessible training of efficient networks, accelerating the development of practical applications. The PyTorch code is available at \url{https://github.com/fedeloper/CycleBNN/}

CycleBNN: Cyclic Precision Training in Binary Neural Networks

TL;DR

The CycleBNN is introduced, an innovative methodology integrating BNNs with cyclic precision training, designed to enhance training efficiency while minimizing the loss in performance, and a path towards faster, more accessible training of efficient networks, accelerating the development of practical applications.

Abstract

This paper works on Binary Neural Networks (BNNs), a promising avenue for efficient deep learning, offering significant reductions in computational overhead and memory footprint to full precision networks. However, the challenge of energy-intensive training and the drop in performance have been persistent issues. Tackling the challenge, prior works focus primarily on task-related inference optimization. Unlike prior works, this study offers an innovative methodology integrating BNNs with cyclic precision training, introducing the CycleBNN. This approach is designed to enhance training efficiency while minimizing the loss in performance. By dynamically adjusting precision in cycles, we achieve a convenient trade-off between training efficiency and model performance. This emphasizes the potential of our method in energy-constrained training scenarios, where data is collected onboard and paves the way for sustainable and efficient deep learning architectures. To gather insights on CycleBNN's efficiency, we conduct experiments on ImageNet, CIFAR-10, and PASCAL-VOC, obtaining competitive performances while using 96.09\% less operations during training on ImageNet, 88.88\% on CIFAR-10 and 96.09\% on PASCAL-VOC. Finally, CycleBNN offers a path towards faster, more accessible training of efficient networks, accelerating the development of practical applications. The PyTorch code is available at \url{https://github.com/fedeloper/CycleBNN/}
Paper Structure (13 sections, 12 equations, 5 figures, 4 tables)

This paper contains 13 sections, 12 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Cyclic Precision in Binary Neural Networks
  4. Binary Convolution
  5. Equalization of Weights
  6. Cyclic Precision during Training
  7. Experiments
  8. Evaluation Metrics
  9. Datasets
  10. Experimental Setup
  11. Results
  12. Ablation Study
  13. Conclusions

Figures (5)

  • Figure 2: An example of cyclic precision scheduling, following Eq. \ref{['scheduling']}, with adjustments between 2-6 bits, across a total of 600 epochs, distributed over 8 cycles
  • Figure 3: Loss landscape visualization after convergence of ResNet-18 (with W/A 1 bit) on CIFAR-10 trained with different precision schedules, where wider contours with larger intervals indicate a better local minima and a lower generalization error as analyzed in li2018visualizing.
  • Figure : (a)
  • Figure : (a)
  • Figure : (b)