Nearest Neighbor Representations of Neural Circuits

TL;DR

The paper addresses representing Boolean circuits using Nearest Neighbor representations inspired by brain-like computation. It develops explicit NN constructions for depth-2 threshold circuits with symmetric or DOM top gates, deriving anchor-based NN representations for functions including convex polytopes, IP2, PARITY, LDLs, and EDLs, with rigorous size and resolution bounds. Key contributions include reflection-based geometric methods to realize convex-polytope membership with $m+1$ anchors, and tailored anchor-placement schemes for LDL/EDL circuits yielding $(m+1)$ or $(m+1)2^m$ anchors and corresponding resolutions, along with special cases like IP2 and OMB. These results strengthen the link between NN representations and neural-network-inspired computation, offering concrete tools for analyzing NN complexity and guiding brain-inspired computation, while pointing to challenges in extending to LT∘LT, removing regularity, and exploring deeper architectures.

Abstract

Neural networks successfully capture the computational power of the human brain for many tasks. Similarly inspired by the brain architecture, Nearest Neighbor (NN) representations is a novel approach of computation. We establish a firmer correspondence between NN representations and neural networks. Although it was known how to represent a single neuron using NN representations, there were no results even for small depth neural networks. Specifically, for depth-2 threshold circuits, we provide explicit constructions for their NN representation with an explicit bound on the number of bits to represent it. Example functions include NN representations of convex polytopes (AND of threshold gates), IP2, OR of threshold gates, and linear or exact decision lists.

Figures (7)

• Figure 1: An illustration of an Retrieval Augmented Generation (RAG) pipeline in NLP lewis2020retrieval. Each "document", which can be a single sentence, a book, or an image of a cat, is converted into a high-dimensional embedding by Embedding Models and is stored in a vector database. To do knowledge-intensive tasks, similar "documents" can be found by approximate NN search algorithms.
• Figure 2: NN representations for $2$-input Boolean functions $\text{AND}(X_1,X_2)$ (left) and $\text{OR}(X_1,X_2)$ (right). Triangles denote $f(X) = 1$ and squares denote $f(X) = 0$. It can be seen that red anchors are closest to squares and blue anchors are closest to triangles. A separating line between anchors pairs is drawn.
• Figure 3: A Linear Decision List of Depth 3 with $5$ binary inputs.
• Figure 4: The depth-2 threshold circuit construction of a LDL $l(X)$ with a DOM gate on top. This shows that $l(X) \in \text{LT}\circ\text{LT}$. The signs of the powers of two depends on the labeling of the outputs $z_i$s. If the first layer consists of exact threshold gates, then we have an EDL.
• Figure 5: A convex polytope defined by the intersection of half-spaces $Ax \leq b$ and its NN representation by the "reflection" argument. The interior of the polytope is closest to $a_0$ and the exterior is closest to one of $a_1,\dots,a_5$.

Theorems & Definitions (22)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Theorem 1
• proof
• Corollary 1.1
• Theorem 2
• proof
• Corollary 2.1