Fault-Tolerant Neural Networks from Biological Error Correction Codes

TL;DR

The paper tackles whether reliable computation is possible with unreliable neurons by adapting biologically observed grid-code error-correction to neural networks, giving a theory of failure thresholds for both digital and analog noise. It proves a neural-network fault-tolerance framework: digital errors from synaptic failure can be mitigated via concatenated repetition codes, while analog Gaussian noise can be suppressed using grid-code–based encoding with a decode-then-encode pipeline to achieve universal computation. A key result is a scaling law showing the number of physical neurons needed per logical neuron grows as $\mathcal{O}(e^{\beta \sigma^2} \log(1/\epsilon))$, enabling polylogarithmic overhead for reliable circuits, and a threshold analysis under combined Gaussian noise and synaptic failure demonstrates a sharp transition between fault-tolerant and faulty regimes with thresholds $p_0$ and $\sigma_0$. The work also demonstrates concrete constructions, including a fault-tolerant neural NAND gate and a two-bit multiplier, and discusses how grid-code–based redundancy could reflect biological realities, suggesting a mechanism for reliable cortex computations and informing neuromorphic hardware design.

Abstract

It has been an open question in deep learning if fault-tolerant computation is possible: can arbitrarily reliable computation be achieved using only unreliable neurons? In the grid cells of the mammalian cortex, analog error correction codes have been observed to protect states against neural spiking noise, but their role in information processing is unclear. Here, we use these biological error correction codes to develop a universal fault-tolerant neural network that achieves reliable computation if the faultiness of each neuron lies below a sharp threshold; remarkably, we find that noisy biological neurons fall below this threshold. The discovery of a phase transition from faulty to fault-tolerant neural computation suggests a mechanism for reliable computation in the cortex and opens a path towards understanding noisy analog systems relevant to artificial intelligence and neuromorphic computing.

Figures (7)

• Figure 1: Schematic comparison of error correction and fault-tolerance. While error correction uses noiseless gates to correct errors (red crosses), fault-tolerance must use faulty gates to generate reliable output. Note that errors on states in the fault-tolerant setup can be rewritten as errors on gates, i.e., faulty wires do not have to be directly considered.
• Figure 2: The recursive concatenation scheme, based on a ternary repetition code, used to construct a logical nand gate at concatenation level $\ell+1$ (denoted nand$^{(\ell+1)}_p$) from logical nand gates at concatenation level $\ell$, with the base case $\textsc{nand}^{(0)}_p = \textsc{nand}_p$. The gates denoted $\text{MAJ}^{(\ell)}_p$ indicate a majority voting operation built from $\textsc{nand}^{(\ell)}_p$ gates, whose explicit construction is illustrated in the inset.
• Figure 3: (A) The recursive concatenation scheme of digital fault-tolerance is extended to construct a logical ReLU at concatenation level $\ell+1$ from logical ReLUs at level $\ell$. Note how the fault-tolerant ReLU is a generalization of the fault tolerant NAND gate in \ref{['fig:NAND_Concatenate']}. The gates denoted $\text{Med}^{(\ell)}_p$ indicate a median operation that is composed of ReLU$^{(\ell)}_p$'s and used to correct errors; its explicit construction is presented in \ref{['eq:Median']}. This construction ultimately generates a logical neuron for a fault-tolerant neural network in the presence of synaptic failure. (B) The logical error probability of ReLU$^{(\ell)}_p(x)$ on random inputs $x \in [-1,1]$ as determined by numerical simulation. The pseudothresholds (red crosses) occur when the error probability intersects that of $\ell=0$; they converge exponentially to the threshold $p_0 \approx 3.72\%$ (vertical black line) with increasing $\ell$.
• Figure 4: (A) Biological setting of the grid code. Neuron firings form a hexagonal lattice with different spacings $\lambda_j$, with lattice sites corresponding to physical locations of an animal in the lab. (B) Example encoding performed by the grid code over $M=15$ moduli $\{\lambda_1, \dots, \lambda_{15}\}$. Observe that these phases are well-approximated as being drawn uniformly at random, in accordance with the formalism of the grid code. (C) Example decoding of phases representing $x=0.5$. The possible decodings allowed by a given phase (indicated by a unique color for each $\lambda_j$) are periodic. Each decoded phase is subject to Gaussian noise (inset). Since the phases constructively add at the true decoded value, maximum likelihood estimation selects the value with the highest signal.
• Figure 5: (A) Logical neuron decomposed into physical neurons to achieve fault-tolerance in the presence of analog noise. The neuron receives encoded neural outputs from the previous layer and performs a computation with time advancing to the right. The logical weights $a_i$ are applied in the codespace, and the decoder recovers $x_k$ by performing error correction. The encoder (red) performs a logical activation function (e.g., ReLU) using appropriate weights and encodes back to the codespace. (B) The logical ReLU encoding function $e(x_i)$ used in the encoder (\ref{['eq:relu']}) for $X' = X/2$. This function is a ReLU repeated over $X/X' = 2$ periods; the construction allows the fault-tolerant neural network to implement the standard ReLU activation function.