Towards Narrowing the Generalization Gap in Deep Boolean Networks

TL;DR

The paper tackles the generalization gap of deep Boolean networks by introducing locality-preserving sampling and logical skip connections, enabling deeper architectures that retain spatial information and enjoy improved gradient flow. It formulates Deep Boolean Networks with hierarchical composition of Boolean functions, learnable gate mixtures, and a voting-based output, trained end-to-end. Empirical results on CIFAR-10/100 and STL-10 show DBNs achieving competitive accuracy with far fewer parameters than several baselines, with data augmentation and depth further boosting performance. The work highlights hardware-friendly, energy-efficient deep learning as a viable direction, with potential extensions to attention mechanisms and larger model families.

Abstract

The rapid growth of the size and complexity in deep neural networks has sharply increased computational demands, challenging their efficient deployment in real-world scenarios. Boolean networks, constructed with logic gates, offer a hardware-friendly alternative that could enable more efficient implementation. However, their ability to match the performance of traditional networks has remained uncertain. This paper explores strategies to enhance deep Boolean networks with the aim of surpassing their traditional counterparts. We propose novel methods, including logical skip connections and spatiality preserving sampling, and validate them on vision tasks using widely adopted datasets, demonstrating significant improvement over existing approaches. Our analysis shows how deep Boolean networks can maintain high performance while minimizing computational costs through 1-bit logic operations. These findings suggest that Boolean networks are a promising direction for efficient, high-performance deep learning models, with significant potential for advancing hardware-accelerated AI applications.

Figures (5)

• Figure 1: In existing literature, optimizing deep neural networks often requires techniques like pruning and quantization when using floating-point precision. In contrast, boolean logic networks operate directly with binary expressions, potentially simplifying hardware implementation without the need for additional methods. This aligns with the principles of Hardware-friendly AI, where logic minimization/optimization can be conducted concurrently, including technology-independent or technology-dependent logic design and optimization based on specific logic libraries.
• Figure 2: Proposed Deep Boolean Networks (DBNs) include locality-preserving sampling, compositional Boolean functions, logical skip connections, and a voting layer. Further details are provided in Figures \ref{['fig:locality_sampling']} and \ref{['fig:skip-connection']}.
• Figure 3: Comparison among (a) convolution, (b) inner-product, and (c) logical (boolean) operations for producing one output from an image input $\mathbb{R}^{C_{in} \times h \times w}$: (a) Convolution uses $C_{in} \cdot k^2$ floating-point parameters, depending on input channels $C_{in}$ and kernel size $k^2$. (b) Inner-product in MLPs uses $d = C_{in} \cdot h \cdot w$ parameters, based on input dimensionality. (c) Logical (boolean) operation requires only two or three boolean operations, in contrast to the vector multiplications in convolutions and MLPs. This is simplified to a Lookup Table (LUT) during inference.
• Figure 4: Examples of locality-preserving sampling in images, collecting pairwise elements from adjacent locations around a base point. Examples of pairing when the base points are $x_0$ and $x_1$ respectively.
• Figure 5: Comparison of skip connections in MLPs and Boolean networks. Unlike the fixed summation ($+$) in MLPs, boolean networks employ various logical operators can be used for the skip connection.

Theorems & Definitions (4)

• Definition 3.1
• Definition 3.2
• Definition 3.3
• Definition 3.4