Nearest Neighbor Representations of Neurons

TL;DR

The paper investigates the NN Representation of threshold functions, focusing on the trade-off between the number of anchors and the resolution of their coordinates. It proves that exact threshold functions have high-resolution, small-anchor representations (up to 3 anchors with $O(\mathrm{RES}(\|w\|_2^2))$ resolution for non-linear cases, and 2 anchors with $O(\mathrm{RES}(\|w\|_2))$ for linear ones, overall $O(n\log n)$ in general) and then constructs low-resolution NN representations for EQ, COMP, and OMB with polynomially many anchors and $O(\log n)$ resolution in various configurations. The main contributions include explicit low-resolution constructions for EQ$_{2n}$ (varying sizes and resolutions), and near-optimal anchors for COMP$_{2n}$ and OMB$_n$, plus a broader open question about all threshold functions. These results offer insight into brain-inspired encoding by showing how more anchors can reduce precision requirements while preserving representational power.

Abstract

The Nearest Neighbor (NN) Representation is an emerging computational model that is inspired by the brain. We study the complexity of representing a neuron (threshold function) using the NN representations. It is known that two anchors (the points to which NN is computed) are sufficient for a NN representation of a threshold function, however, the resolution (the maximum number of bits required for the entries of an anchor) is $O(n\log{n})$. In this work, the trade-off between the number of anchors and the resolution of a NN representation of threshold functions is investigated. We prove that the well-known threshold functions EQUALITY, COMPARISON, and ODD-MAX-BIT, which require 2 or 3 anchors and resolution of $O(n)$, can be represented by polynomially large number of anchors in $n$ and $O(\log{n})$ resolution. We conjecture that for all threshold functions, there are NN representations with polynomially large size and logarithmic resolution in $n$.

Figures (4)

• Figure 1: NN representations for $2$-input Boolean functions $\text{AND}(X_1,X_2)$ (left) and $\text{XOR}(X_1,X_2)$ (right). Triangles denote $f(X) = 1$ and squares denote $f(X) = 0$. It can be seen that red anchors are closest to squares and blue anchors are closest to triangles. Separating lines between anchors pairs are drawn.
• Figure 2: The NN Representation of a linear threshold function $\mathds{1}\{w^T X \geq b\}$ and its $2$-anchor NN Representation. $X^*$ can be any point in the hyperplane.
• Figure 3: The NN Representation of an Exact Threshold Function $\mathds{1}\{w^T X = b\}$ and its $3$-anchor NN Representation. The anchors $a_1$ and $a_2$ must be close enough to the hyperplane. All anchors are collinear.
• Figure 4: The construction idea for the $\text{COMP}(X,Y)$ function depicting the first two iterations. For $2n$-inputs, there will be $n$ iterations resulting in $2n$ many anchors.

Theorems & Definitions (17)

• Definition 1
• Definition 2
• Definition 3
• Theorem 1: kilic2023information
• Definition 4
• Definition 5
• Theorem 2
• proof
• Definition 6
• Theorem 3