Entropic Efficiency of Bayesian Inference Protocols

TL;DR

This work frames Bayesian inference as a thermodynamic cycle of measure, infer, and erase, introducing inferential efficiency $\eta = \mathcal{I}/\mathcal{C}$ that bounds how effectively information gained translates into memory erasure cost. It analyzes two memory-access paradigms—sequential (temporal correlations) and parallel (spatial correlations)—and shows that, when all system–memory correlations are exploited, both achieve the same minimal erasure cost even in the presence of noise; otherwise, parallel schemes can outperform sequential ones by leveraging cross-memory correlations. The authors provide a concrete binary-bit example with analytic bounds on information gain and erasure costs, illustrating how correlation structure and noise shape efficiency. The results offer a physically grounded criterion for comparing inference strategies and have implications for metrology, tomography, and energy-aware machine learning by linking target information gains to entropic costs.

Abstract

Inference is a versatile tool that underlies scientific discovery, machine learning, and everyday decision-making: it describes how an agent updates a probability distribution as partial information is acquired from multiple measurements, reducing ignorance about a system's latent state. We define an inferential efficiency as the ratio of information gain to cumulative memory erasure cost, with inefficiency arising from unexploited correlations between the measured system and memories, and/or between memories and environment (noise). Using this efficiency, we benchmark two limiting measurement paradigms: sequential, in which the same memory is exploited iteratively, and parallel, in which many memories are exploited simultaneously. In both cases, the minimal erasure cost reflects correlations across memories: temporal in sequential, spatial in parallel. Remarkably, when all system-memory correlations are exploited for inference, both paradigms attain the same minimal erasure cost, even in the presence of noise. Conversely, the parallel paradigm performs better in the presence of unexploited correlations, stemming from hidden memories' degrees of freedom. This approach provides a quantitative, physically grounded criterion to compare inference strategies, determine their efficiency, and link target information gains to their minimal entropic cost.

Figures (7)

• Figure 1: (a) Autonomous measure–infer–erase cycle. An agent provides the system $S$ together with a prior $p_S^\mathtt{0}(x)$, which is processed by an autonomous machine equipped with a structured memory $M=(Q,R)$, and environment $E$, and a thermal bath $B$. Measurement via an entropy-preserving map $\mathcal{M}$ generates correlations in $Q$ and a coarse-grained outcome $r_0$ in $R$, which is then processed by a reversible Bayesian update to yield $p_S^\mathtt{1}(x)$. An optimal erasure protocol subsequently resets the memory through coupling to the thermal bath. (b) Sequential architecture ($n=3$), in which the measure-infer-erase cycle is iterated: the posterior from each cycle becomes the prior for the next, enabling temporal correlations during erasure. (c) Parallel architecture ($n=3$), realized by extending the autonomous machine to include multiple memories that record outcomes concurrently and are erased simultaneously, enabling spatial correlations during erasure. Together, the sequential and parallel paradigms exhibit complementary trade-offs between temporal resources and hardware complexity.
• Figure 2: Illustrative example of inferential efficiency for multiple measurements of a binary system. (a) The information gain $\mathcal{I}(n)$ increases rapidly for small $\varepsilon$, reaching near-maximal certainty for modest $n$. (b) The cost of a single-cycle $\mathcal{C}_0$, similarly increases as $\varepsilon$ decreases; crucially $\mathcal{C}_0 > \mathcal{I}(1)$ for all $\varepsilon \in (0,1/2)$, reflecting the inefficiency arising from unexploited system-memory correlations. (c) Entropic efficiency of the three paradigms as a function of $n$: parallel approaches unity, sequential plateaus to a finite value, and uncorrelated decays to zero. (d,e) Information gain versus erasure cost (metric-resource representation) for sequential and parallel strategies at $\varepsilon=0.1$ and $\varepsilon=0.25$ , respectively, illustrating the widening efficiency gap as noise increases.
• Figure S1: Erasure operations optimized for distinct memory statistics, implemented via coupling to a thermal bath and leading to the same reset state with different minimal costs.
• Figure S2: Venn-diagram representation of the joint entropy of a system $S$ and two memories $M_0$ and $M_1$, together with their respective registers $R_0 \subset M_0$ and $R_1 \subset M_1$ used for inference. The area of each region represents the entropy of the corresponding subsystem, while the overlap between any two regions $A$ and $B$ represents their mutual information $I(A : B)$.
• Figure S3: Expected entropy reduction on $S$ through correlations with $R_0$ and $R_1$.