Faster Predictive Coding Networks via Better Initialization

TL;DR

This work tackles the computational bottleneck of predictive coding networks (PCNs) by focusing on neuron initialization to reduce iterative inference. It introduces two complementary strategies: stream-aligned average initialization (I$_{\text{avg}}$) to stabilize hidden states during classification and memory-based initialization (I$_{\text{mem}}$) using Hopfield networks for unsupervised tasks, plus a hybrid approach combining forward initialization for early layers with average initialization for deeper layers. Empirically, I$_{\text{avg}}$ delivers faster convergence and higher accuracy across multiple supervised benchmarks, approaching or even surpassing backpropagation in some configurations, while I$_{\text{mem}}$ yields superior performance and stability in unsupervised generation. Overall, the results demonstrate substantial reductions in sequential matrix multiplications (SMMs) and training time, suggesting a viable path for making neuro-inspired energy-based learning competitive with conventional deep learning on GPUs.

Abstract

Research aimed at scaling up neuroscience inspired learning algorithms for neural networks is accelerating. Recently, a key research area has been the study of energy-based learning algorithms such as predictive coding, due to their versatility and mathematical grounding. However, the applicability of such methods is held back by the large computational requirements caused by their iterative nature. In this work, we address this problem by showing that the choice of initialization of the neurons in a predictive coding network matters significantly and can notably reduce the required training times. Consequently, we propose a new initialization technique for predictive coding networks that aims to preserve the iterative progress made on previous training samples. Our approach suggests a promising path toward reconciling the disparities between predictive coding and backpropagation in terms of computational efficiency and final performance. In fact, our experiments demonstrate substantial improvements in convergence speed and final test loss in both supervised and unsupervised settings.

Figures (10)

• Figure 1: Discriminative PCN with $4$ layers. For supervised training, we fix the first and last layers to the input-output pair $(x,y)$.
• Figure 2: Performance of different initialization methods. (Left) Training curves with test accuracy over epochs (solid lines) and SMMs (dotted lines, plotted until the highest accuracy is reached for each method). The shaded area corresponds to the standard deviation over 3 seeds. (Right) Impact of the number of inference steps $T$. Only $\mathcal{I}_{\text{fw}}$ is able to train the model at low values of $T$.
• Figure 3: Comparison between batch training and stream aligned training on a classification task. Different colors represent different classes. (Left) Batch training: samples are randomly shuffled within a batch; consequently the hidden state of the network after $T$ steps cannot be preserved, as new samples may belong to different classes, which would result in high error values in the first and last layers. (Right) Stream-aligned training: the hidden state is always associated with elements of the same class, reducing the data distribution variance and, consequently, the error. Furthermore, by taking class-wise averages of the hidden neurons, we further spread the energy through the network, as highlighted by the red-shaded lines (saturation represents error).
• Figure 4: PCN whose layers are enhanced with associative memories for better initialization.
• Figure 5: Test accuracy for different values of inference steps $T$ used during training. Shades of color indicates different fractions of training data used (i.e., $25\%$ is lighter; $50\%$, $100\%$ are darker).

Theorems & Definitions (2)

• Definition 3.1
• Proposition 3.2