Learning Compact Boolean Networks

TL;DR

This work tackles the challenge of learning compact Boolean networks by addressing three core aspects: efficient, parameter-free connection learning; a compact convolutional Boolean architecture that avoids tree-based kernels; and discretization-aware training via progressive layer-wise discretization. The authors introduce a novel per-neuron parameterization with adaptive resampling based on weight-entropy to learn connections without extra parameters, and demonstrate that a single-operation convolutional kernel can achieve state-of-the-art accuracy with far fewer Boolean operations than tree-based designs. Adaptive discretization further reduces the discretization gap and speeds training, enabling high accuracy on MNIST and CIFAR-10 with up to 37x fewer Boolean operations than prior methods. Overall, the approach yields Pareto improvements in accuracy versus computation, showing strong potential for resource-constrained deployment of Boolean networks in vision tasks.

Abstract

Floating-point neural networks dominate modern machine learning but incur substantial inference cost, motivating interest in Boolean networks for resource-constrained settings. However, learning compact and accurate Boolean networks is challenging due to their combinatorial nature. In this work, we address this challenge from three different angles: learned connections, compact convolutions and adaptive discretization. First, we propose a novel strategy to learn efficient connections with no additional parameters and negligible computational overhead. Second, we introduce a novel convolutional Boolean architecture that exploits the locality with reduced number of Boolean operations than existing methods. Third, we propose an adaptive discretization strategy to reduce the accuracy drop when converting a continuous-valued network into a Boolean one. Extensive results on standard vision benchmarks demonstrate that the Pareto front of accuracy vs. computation of our method significantly outperforms prior state-of-the-art, achieving better accuracy with up to 37x fewer Boolean operations.

Figures (10)

• Figure 1: Comparison between our method and prior works on CIFAR-10. Note the log scale on x-axis.
• Figure 2: Left: a Boolean circuit. Right: a Boolean network.
• Figure 3: Comparison between the parameterization with fixed connections (literature) and the proposed parameterization.
• Figure 4: Convergence speed of different layers.
• Figure 5: The result of learning partial connections on CIFAR-10. Note that the x-axis is unequally spaced.