Predictive Coding Graphs are a Superset of Feedforward Neural Networks

TL;DR

It is proved how PCGs define a mathematical superset of feedforward artificial neural networks (multilayer perceptrons), which positions PCNs more strongly within contemporary machine learning (ML), and reinforces earlier proposals to study the use of non-hierarchical neural networks for ML tasks, and more generally the notion of topology in neural networks.

Abstract

Predictive coding graphs (PCGs) are a recently introduced generalization to predictive coding networks, a neuroscience-inspired probabilistic latent variable model. Here, we prove how PCGs define a mathematical superset of feedforward artificial neural networks (multilayer perceptrons). This positions PCNs more strongly within contemporary machine learning (ML), and reinforces earlier proposals to study the use of non-hierarchical neural networks for ML tasks, and more generally the notion of topology in neural networks.

Figures (3)

• Figure 1: PCGs trained with IL generalize the structure of FNNs to arbitrary graphs, including loops and non-hierarchical structures, which are not trainable using BP.
• Figure 2: A PCG weight matrix partitioned into block matrices ($N=10$ nodes, partitioned by 4 layers with 2, 3, 3, and 2 nodes/layer, respectively). In traditional ANNs trained with BP, only feedforward connections (blue and red blocks) are trainable, whereas the full matrix is trainable with IL. This figure extends fig. 4 in salvatori_learning_2022.
• Figure 3: Mapping between PCN indices (blue nodes) and PCG indices (below nodes). Note that layers may have different widths $n_\ell$. $i$ denotes PCN index within a layer, and $I_\ell$ is defined by \ref{['eq:index_subsets']}. One weight is shown in orange using PCN indices (top) and PCG indices (bottom), cf. \ref{['eq:hierarchical_pcg']}.

Theorems & Definitions (7)

• Definition 1
• Definition 2
• Theorem 1
• Definition 3
• Theorem 2
• proof
• proof