Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bayesian Predictive Coding

Alexander Tschantz, Magnus Koudahl, Hampus Linander, Lancelot Da Costa, Conor Heins, Jeff Beck, Christopher Buckley

TL;DR

Bayesian Predictive Coding (BPC) extends predictive coding by placing a Bayesian posterior over neural network parameters, enabling explicit uncertainty quantification while preserving PC's locality through Hebbian-like updates. By using a Matrix Normal-Wishart prior, BPC achieves closed-form updates for posterior parameters, improving convergence in full-batch settings and remaining competitive in mini-batch regimes. Empirical results show that BPC maintains competitive accuracy relative to PC and backpropagation, and it provides robust epistemic and aleatoric uncertainty estimates, outperforming some Bayesian baselines on several tasks. The work suggests that biologically plausible Bayesian learning can support uncertainty-aware deep learning and motivates future work on scalable posterior approximations and uncertainty propagation in large networks.

Abstract

Predictive coding (PC) is an influential theory of information processing in the brain, providing a biologically plausible alternative to backpropagation. It is motivated in terms of Bayesian inference, as hidden states and parameters are optimised via gradient descent on variational free energy. However, implementations of PC rely on maximum \textit{a posteriori} (MAP) estimates of hidden states and maximum likelihood (ML) estimates of parameters, limiting their ability to quantify epistemic uncertainty. In this work, we investigate a Bayesian extension to PC that estimates a posterior distribution over network parameters. This approach, termed Bayesian Predictive coding (BPC), preserves the locality of PC and results in closed-form Hebbian weight updates. Compared to PC, our BPC algorithm converges in fewer epochs in the full-batch setting and remains competitive in the mini-batch setting. Additionally, we demonstrate that BPC offers uncertainty quantification comparable to existing methods in Bayesian deep learning, while also improving convergence properties. Together, these results suggest that BPC provides a biologically plausible method for Bayesian learning in the brain, as well as an attractive approach to uncertainty quantification in deep learning.

Bayesian Predictive Coding

TL;DR

Bayesian Predictive Coding (BPC) extends predictive coding by placing a Bayesian posterior over neural network parameters, enabling explicit uncertainty quantification while preserving PC's locality through Hebbian-like updates. By using a Matrix Normal-Wishart prior, BPC achieves closed-form updates for posterior parameters, improving convergence in full-batch settings and remaining competitive in mini-batch regimes. Empirical results show that BPC maintains competitive accuracy relative to PC and backpropagation, and it provides robust epistemic and aleatoric uncertainty estimates, outperforming some Bayesian baselines on several tasks. The work suggests that biologically plausible Bayesian learning can support uncertainty-aware deep learning and motivates future work on scalable posterior approximations and uncertainty propagation in large networks.

Abstract

Predictive coding (PC) is an influential theory of information processing in the brain, providing a biologically plausible alternative to backpropagation. It is motivated in terms of Bayesian inference, as hidden states and parameters are optimised via gradient descent on variational free energy. However, implementations of PC rely on maximum \textit{a posteriori} (MAP) estimates of hidden states and maximum likelihood (ML) estimates of parameters, limiting their ability to quantify epistemic uncertainty. In this work, we investigate a Bayesian extension to PC that estimates a posterior distribution over network parameters. This approach, termed Bayesian Predictive coding (BPC), preserves the locality of PC and results in closed-form Hebbian weight updates. Compared to PC, our BPC algorithm converges in fewer epochs in the full-batch setting and remains competitive in the mini-batch setting. Additionally, we demonstrate that BPC offers uncertainty quantification comparable to existing methods in Bayesian deep learning, while also improving convergence properties. Together, these results suggest that BPC provides a biologically plausible method for Bayesian learning in the brain, as well as an attractive approach to uncertainty quantification in deep learning.

Paper Structure

This paper contains 19 sections, 27 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methods
  3. Training and testing
  4. Experiments
  5. Accuracy
  6. Uncertainty quantification
  7. Discussion
  8. Distributions
  9. Matrix Normal Distribution
  10. Wishart Distribution
  11. Matrix Normal Wishart Distribution
  12. Derivatives
  13. Predictive Coding
  14. Conjugate updates
  15. Related work
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Accuracy on two classification tasks (two moons and MNIST) and error on one regression task (energy). These results show that BPC converges to the same accuracy in fewer epochs in the full-batch setting (energy and two moons), and is competitive in the mini-batch setting (MNIST). Shaded regions denote 1 standard deviation across 5 seeds. See Appendix \ref{['sec:experiment_details']} for experiment details.
  • Figure 2: Comparison of root mean square error (RMSE) and log predictive density (LPD) metrics on the UCI regression dataset yacht using Bayesian Predictive Coding (BPC) and Bayes by Backprop (BBB). The final two plots show the aleatoric and epistemic uncertainties quantified by the model on synthetic regression tasks. See Appendix \ref{['sec:experiment_details']} for experiment details.