G-Net: A Provably Easy Construction of High-Accuracy Random Binary Neural Networks

TL;DR

The paper tackles constructing high-accuracy random binary neural networks by leveraging hyperdimensional computing and a novel random sign embedding. It builds G-Nets, floating-point networks whose layers can be embedded into binary hypervectors using φ(x)=sign($ nd{G}x$) and Grothendieck's identity, yielding EHD G-Nets with provable concentration-based guarantees. The authors provide non-asymptotic, layer- and network-level consistency bounds, extend the theory to Rademacher embeddings, and demonstrate strong empirical performance on MNIST, CIFAR-10, and other tasks, often surpassing prior HDC models and approaching real-valued CNN performance. The work also analyzes hardware costs, robustness to bit flips, and positions EHD G-Nets as a practical, theory-backed path toward robust binary/quantized deep learning suitable for edge hardware.

Abstract

We propose a novel randomized algorithm for constructing binary neural networks with tunable accuracy. This approach is motivated by hyperdimensional computing (HDC), which is a brain-inspired paradigm that leverages high-dimensional vector representations, offering efficient hardware implementation and robustness to model corruptions. Unlike traditional low-precision methods that use quantization, we consider binary embeddings of data as points in the hypercube equipped with the Hamming distance. We propose a novel family of floating-point neural networks, G-Nets, which are general enough to mimic standard network layers. Each floating-point G-Net has a randomized binary embedding, an embedded hyperdimensional (EHD) G-Net, that retains the accuracy of its floating-point counterparts, with theoretical guarantees, due to the concentration of measure. Empirically, our binary models match convolutional neural network accuracies and outperform prior HDC models by large margins, for example, we achieve almost 30\% higher accuracy on CIFAR-10 compared to prior HDC models. G-Nets are a theoretically justified bridge between neural networks and randomized binary neural networks, opening a new direction for constructing robust binary/quantized deep learning models. Our implementation is available at https://github.com/GNet2025/GNet.

Figures (11)

• Figure 1: A graphical demonstration of data embedding, and bundle embedding
• Figure 2: (a) A G-Net layer (top), and its corresponding EHD G-Net layer (bottom); the input to each block is denoted by $x$ and the output is represented by $y$; (b) A G-Net and corresponding EHD G-Net constructed by cascading the proposed layer blocks
• Figure 3: (a–c) Comparison of Rademacher RASU G-Net with other HDC methods (classic HDC Ge2020Classification, HoloGN ExHoloGN, Laplace HDC LaplaceHDC, OnlineHD onlineHD2021 and RFF-HDCyu2022understanding) on MNIST, CIFAR-10, and WSS (left to right); (d,e) performance of different G-Nets on MNIST (left) and WSS (right)
• Figure 4: The set $S = \left\{ z : u^\top z \le 0 \text{ and } v^\top z \ge 0 \right\}$
• Figure 5: Minor weight distribution shift for wide layers: the distribution of the weights of a fully connected G-Net Layer at epochs $0$, $1$, $10$, and $25$

Theorems & Definitions (61)

• Definition 2.1
• Proposition 2.1
• Lemma 3.1: Grothendieck's Identity
• Definition 3.1: G-Net
• Definition 3.2: EHD G-Net: Embedded Hyperdimensional G-Net
• Theorem 4.1
• Corollary 4.1
• Theorem 4.2
• Definition 4.1
• Theorem 4.3