Convolutional Differentiable Logic Gate Networks

TL;DR

This work extends the approach for learning logic gate networks directly via a differentiable relaxation by deep logic gate tree convolutions, logical OR pooling, and residual initializations, which allows scaling logic gate networks up by over one order of magnitude and utilizing the paradigm of convolution.

Abstract

With the increasing inference cost of machine learning models, there is a growing interest in models with fast and efficient inference. Recently, an approach for learning logic gate networks directly via a differentiable relaxation was proposed. Logic gate networks are faster than conventional neural network approaches because their inference only requires logic gate operators such as NAND, OR, and XOR, which are the underlying building blocks of current hardware and can be efficiently executed. We build on this idea, extending it by deep logic gate tree convolutions, logical OR pooling, and residual initializations. This allows scaling logic gate networks up by over one order of magnitude and utilizing the paradigm of convolution. On CIFAR-10, we achieve an accuracy of 86.29% using only 61 million logic gates, which improves over the SOTA while being 29x smaller.

Figures (11)

• Figure 1: Gate count vs. accuracy plot on the CIFAR-10 data set. Our models ($\bigstar$) are substantially above the pareto-front of the SOTA baselines. Gate counts are proportional to chip area. Our models are more efficient than the SOTA by factors of $\geq29\times$. Note that the $x$-axis (gate count) is on a log-scale.
• Figure 2: Architecture of a randomly connected LGN. Each node corresponds to one logic gate. During training, the distribution over choices of logic gates (bottom, 16 options) is learned for each node.
• Figure 3: Conventional convolutional neural networks (a) compared to convolutional logic gate networks (b). The images illustrate the first and second to last kernel placements. The nodes correspond to weighted sums (a), and binary logic gates $f_1, f_2, f_3$ (b), respectively. The weights / choices of logic gates are shared between kernel placements. For visual simplicity, only a single input channel and kernel (output channel) is displayed.
• Figure 4: Plot of the density of activations for the second convolutional block of an or-pooling based convolutional LGN. It shows that training implicitly enforces that the outputs of the block have the activation level of a no-pooling network (i.e., with pure stride).
• Figure 5: Gaussian initialization (a) petersen2022difflogic vs. our residual initialization (b).