Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BiPer: Binary Neural Networks using a Periodic Function

Edwin Vargas, Claudia Correa, Carlos Hinojosa, Henry Arguello

TL;DR

This work introduces BiPer, a binary periodic function for binarizing neural network weights in Binary Neural Networks (BNNs) to curb gradient mismatch and quantization error. By forward-pass binarization with a square wave $w^q = \operatorname{Sign}(\sin(\omega_0 w))$ and a differentiable backward surrogate $\hat{w}=\sin(\omega_0 w)$, BiPer enables gradient flow with magnitude scaled by the frequency $\omega_0$, allowing explicit QE control. The authors provide a QE analysis showing a maximal QE of $0.102835$ at $b\omega_0\approx0.954882$ and validate the method on CIFAR-10 and ImageNet, demonstrating up to 1% and 0.63% improvements over state-of-the-art BNNs, respectively, with a two-stage training protocol. The approach yields competitive results across architectures (ResNet/ VGG-Small) and datasets, highlighting the practical impact of frequency-aware binarization for resource-constrained deployment and paving the way for further exploration of periodic-function-based quantization in neural networks.

Abstract

Quantized neural networks employ reduced precision representations for both weights and activations. This quantization process significantly reduces the memory requirements and computational complexity of the network. Binary Neural Networks (BNNs) are the extreme quantization case, representing values with just one bit. Since the sign function is typically used to map real values to binary values, smooth approximations are introduced to mimic the gradients during error backpropagation. Thus, the mismatch between the forward and backward models corrupts the direction of the gradient, causing training inconsistency problems and performance degradation. In contrast to current BNN approaches, we propose to employ a binary periodic (BiPer) function during binarization. Specifically, we use a square wave for the forward pass to obtain the binary values and employ the trigonometric sine function with the same period of the square wave as a differentiable surrogate during the backward pass. We demonstrate that this approach can control the quantization error by using the frequency of the periodic function and improves network performance. Extensive experiments validate the effectiveness of BiPer in benchmark datasets and network architectures, with improvements of up to 1% and 0.69% with respect to state-of-the-art methods in the classification task over CIFAR-10 and ImageNet, respectively. Our code is publicly available at https://github.com/edmav4/BiPer.

BiPer: Binary Neural Networks using a Periodic Function

TL;DR

This work introduces BiPer, a binary periodic function for binarizing neural network weights in Binary Neural Networks (BNNs) to curb gradient mismatch and quantization error. By forward-pass binarization with a square wave and a differentiable backward surrogate , BiPer enables gradient flow with magnitude scaled by the frequency , allowing explicit QE control. The authors provide a QE analysis showing a maximal QE of at and validate the method on CIFAR-10 and ImageNet, demonstrating up to 1% and 0.63% improvements over state-of-the-art BNNs, respectively, with a two-stage training protocol. The approach yields competitive results across architectures (ResNet/ VGG-Small) and datasets, highlighting the practical impact of frequency-aware binarization for resource-constrained deployment and paving the way for further exploration of periodic-function-based quantization in neural networks.

Abstract

Quantized neural networks employ reduced precision representations for both weights and activations. This quantization process significantly reduces the memory requirements and computational complexity of the network. Binary Neural Networks (BNNs) are the extreme quantization case, representing values with just one bit. Since the sign function is typically used to map real values to binary values, smooth approximations are introduced to mimic the gradients during error backpropagation. Thus, the mismatch between the forward and backward models corrupts the direction of the gradient, causing training inconsistency problems and performance degradation. In contrast to current BNN approaches, we propose to employ a binary periodic (BiPer) function during binarization. Specifically, we use a square wave for the forward pass to obtain the binary values and employ the trigonometric sine function with the same period of the square wave as a differentiable surrogate during the backward pass. We demonstrate that this approach can control the quantization error by using the frequency of the periodic function and improves network performance. Extensive experiments validate the effectiveness of BiPer in benchmark datasets and network architectures, with improvements of up to 1% and 0.69% with respect to state-of-the-art methods in the classification task over CIFAR-10 and ImageNet, respectively. Our code is publicly available at https://github.com/edmav4/BiPer.
Paper Structure (22 sections, 19 equations, 4 figures, 4 tables)

This paper contains 22 sections, 19 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Binary Neural Networks
  4. Periodic Functions in Deep Neural Networks
  5. BNN Preliminaries
  6. BiPer
  7. Quantization Error Analysis
  8. Experiments
  9. Experiments Setup
  10. Data Sets
  11. CIFAR-10 cifar-10dataset:
  12. ImageNet Imagenet_ILSVRC12:
  13. Network Architectures
  14. Training Details and Procedures
  15. Two stage training:
  16. ...and 7 more sections

Figures (4)

  • Figure 1: (a) Sign function. (b) Binary periodic function.
  • Figure 2: Probability density function of $\hat{w}=\textrm{sin}(\omega_0 w)$ assuming that the random variable $w$ follows a Laplace distribution with parameter $b$ and a fixed value of $\omega_0=1$.
  • Figure 3: Quantization error as a function of the frequency $\omega_0$ for different values of $b$. The proposed BiPer approach is able to control QE with the frequency of the periodic function.
  • Figure 4: Impact of the frequency of the periodic function $\omega_0$ on BiPer (a) Top-1 classification precision, (b) Quantization error and, (c) Weight distribution parameter $b$, for the CIFAR-10 data set.