ePC: Fast and Deep Predictive Coding for Digital Hardware

TL;DR

This work identifies a fundamental inefficiency in state-based Predictive Coding (sPC) arising from exponential signal decay when simulated on digital hardware. It introduces error-based PC (ePC), a reparameterization that uses prediction errors $oldsymbol{ε}$ as the primary optimization variables, yielding global connectivity and eliminating depth-related attenuation while preserving the same equilibrium as sPC, i.e., the same weight gradients at convergence. Empirically, ePC converges orders of magnitude faster than sPC and matches backpropagation performance across deep architectures and multiple datasets, addressing a key barrier to scaling PC on digital hardware. The findings provide a theoretical and practical foundation for scalable PC-based learning and outline avenues for applying PC in neuromorphic contexts and domain-specific learning tasks.

Abstract

Predictive Coding (PC) offers a brain-inspired alternative to backpropagation for neural network training, described as a physical system minimizing its internal energy. However, in practice, PC is predominantly digitally simulated, requiring excessive amounts of compute while struggling to scale to deeper architectures. This paper reformulates PC to overcome this hardware-algorithm mismatch. First, we uncover how the canonical state-based formulation of PC (sPC) is, by design, deeply inefficient in digital simulation, inevitably resulting in exponential signal decay that stalls the entire minimization process. Then, to overcome this fundamental limitation, we introduce error-based PC (ePC), a novel reparameterization of PC which does not suffer from signal decay. Though no longer biologically plausible, ePC numerically computes exact PC weights gradients and runs orders of magnitude faster than sPC. Experiments across multiple architectures and datasets demonstrate that ePC matches backpropagation's performance even for deeper models where sPC struggles. Besides practical improvements, our work provides theoretical insight into PC dynamics and establishes a foundation for scaling PC-based learning to deeper architectures on digital hardware and beyond.

Figures (10)

• Figure 1: Dynamics of layerwise energies in Predictive Coding. Standard state-based PC struggles to propagate signals through the network (from output to input), with progressively longer delays at deeper layers. By contrast, error-based PC converges to equilibrium in just a few update steps, thanks to its global signal propagation. Results for an untrained 20-layer MLP on a random MNIST input.
• Figure 2: Structural comparison of sPC (left) and ePC (right), highlighting functional equivalence
• Figure 3: Step-by-step dynamics of sPC reveals an exponential signal decay in the backward path.
• Figure 4: :State-based PC(standard)
• Figure 5: State convergence dynamics for the bottom, middle, and top hidden layers in a 20-layer linear PC network trained on MNIST. Curves show batch medians (n=64) of L2 distance to the analytical optimum, with interquartile shading. Thanks to its superior signal propagation, ePC converges over $100\times$ faster than sPC, while reaching the exact same equilibrium.

Theorems & Definitions (18)

• Theorem 3.1: Bijective Mapping
• proof
• Theorem 3.2: Energy Equivalence
• proof
• Lemma 3.3: Jacobian Structure
• proof
• Theorem 3.4: Gradient Relationship
• proof
• Theorem 3.5: Critical Point Correspondence
• proof