Bitwise Neural Networks
Minje Kim, Paris Smaragdis
TL;DR
Bitwise Neural Networks replace floating-point representations with bipolar binary parameters, enabling feedforward through bitwise operations like XNOR and sign activation. The authors propose a two-stage training pipeline—weight compression to initialize a binarized model followed by noisy backpropagation on binarized weights and signals—to bridge real-valued networks and fully binary implementations. Experiments on MNIST with binary features demonstrate competitive performance with large computational savings, highlighting the practicality of BNNs for resource-constrained hardware. This work suggests a viable path toward highly efficient neural networks suitable for embedded and low-power devices, while indicating future work toward convolutional extensions.
Abstract
Based on the assumption that there exists a neural network that efficiently represents a set of Boolean functions between all binary inputs and outputs, we propose a process for developing and deploying neural networks whose weight parameters, bias terms, input, and intermediate hidden layer output signals, are all binary-valued, and require only basic bit logic for the feedforward pass. The proposed Bitwise Neural Network (BNN) is especially suitable for resource-constrained environments, since it replaces either floating or fixed-point arithmetic with significantly more efficient bitwise operations. Hence, the BNN requires for less spatial complexity, less memory bandwidth, and less power consumption in hardware. In order to design such networks, we propose to add a few training schemes, such as weight compression and noisy backpropagation, which result in a bitwise network that performs almost as well as its corresponding real-valued network. We test the proposed network on the MNIST dataset, represented using binary features, and show that BNNs result in competitive performance while offering dramatic computational savings.