Benchmarking Predictive Coding Networks -- Made Simple

TL;DR

Predictive coding networks (PCNs) offer a local, energy-based alternative to backpropagation but face scalability challenges in deep hierarchies. The authors introduce PCX, a $JAX$-based library, and a comprehensive benchmarking suite to standardize training, evaluation, and comparisons across discriminative and generative PC settings, including $T$ inference steps and various learning rules. They report state-of-the-art results on benchmarks like CIFAR100 and Tiny ImageNet for PCN variants, particularly with nudging, and analyze energy propagation, initialization, and training stability to identify limitations and future directions. The work provides open-source code, tutorials, and architecture- and dataset-wide analyses to accelerate progress toward scalable PCNs, with broad impact on reproducibility and community-driven benchmarking.

Abstract

In this work, we tackle the problems of efficiency and scalability for predictive coding networks (PCNs) in machine learning. To do so, we propose a library, called PCX, that focuses on performance and simplicity, and use it to implement a large set of standard benchmarks for the community to use for their experiments. As most works in the field propose their own tasks and architectures, do not compare one against each other, and focus on small-scale tasks, a simple and fast open-source library and a comprehensive set of benchmarks would address all these concerns. Then, we perform extensive tests on such benchmarks using both existing algorithms for PCNs, as well as adaptations of other methods popular in the bio-plausible deep learning community. All this has allowed us to (i) test architectures much larger than commonly used in the literature, on more complex datasets; (ii)~reach new state-of-the-art results in all of the tasks and datasets provided; (iii)~clearly highlight what the current limitations of PCNs are, allowing us to state important future research directions. With the hope of galvanizing community efforts towards one of the main open problems in the field, scalability, we release code, tests, and benchmarks. Link to the library: https://github.com/liukidar/pcx

Figures (26)

• Figure 1: (a): Generative and discriminative modes; (b): Pseudocode of PC in supervised learning, where both the latent variables $h_l$ and the weight parameters $\theta_l$ are updated to minimize the variational free energy $\mathcal{F}$. In the colored boxes, informal description of the different algorithms considered in this work.
• Figure 2: Test accuracies of different PC algorithms on the CIFAR10 dataset, using models of different depths.
• Figure 3: CIFAR10 image reconstruction via autoencoding convolutional networks. In order: original, PC, iPC, and BP.
• Figure 4: Generative samples obtained by MCPC. Left: Contour plot of learned generative distribution compared to Iris data samples (x). Right: Samples obtained for a PCN. In order: unconditional generation, conditional generation (odd), conditional generation (even).
• Figure 5: Memory recalled images. Top: Original images. Left: Noisy input (guassian noise, $\sigma=0.2$) and reconstruction. Right: Masked input (bottom half removed) and reconstruction.