Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Thermodynamic Structure of Asymptotic Inference

Willy Wong

TL;DR

This framework suggests that ensemble physics and inferential physics constitute shadow processes evolving in opposite directions within a unified thermodynamic description.

Abstract

A thermodynamic framework for asymptotic inference is developed in which sample size and parameter variance define a state space. Within this description, Shannon information plays the role of entropy, and an integrating factor organizes its variation into a first-law-type balance equation. The framework supports a cyclic inequality analogous to a reversed second law, derived for the estimation of the mean. A non-trivial third-law-type result emerges as a lower bound on entropy set by representation noise. Optimal inference paths, global bounds on information gain, and a natural Carnot-like information efficiency follow from this structure, with efficiency fundamentally limited by a noise floor. Finally, de Bruijn's identity and the I-MMSE relation in the Gaussian-limit case appear as coordinate projections of the same underlying thermodynamic structure. This framework suggests that ensemble physics and inferential physics constitute shadow processes evolving in opposite directions within a unified thermodynamic description.

A Thermodynamic Structure of Asymptotic Inference

TL;DR

This framework suggests that ensemble physics and inferential physics constitute shadow processes evolving in opposite directions within a unified thermodynamic description.

Abstract

A thermodynamic framework for asymptotic inference is developed in which sample size and parameter variance define a state space. Within this description, Shannon information plays the role of entropy, and an integrating factor organizes its variation into a first-law-type balance equation. The framework supports a cyclic inequality analogous to a reversed second law, derived for the estimation of the mean. A non-trivial third-law-type result emerges as a lower bound on entropy set by representation noise. Optimal inference paths, global bounds on information gain, and a natural Carnot-like information efficiency follow from this structure, with efficiency fundamentally limited by a noise floor. Finally, de Bruijn's identity and the I-MMSE relation in the Gaussian-limit case appear as coordinate projections of the same underlying thermodynamic structure. This framework suggests that ensemble physics and inferential physics constitute shadow processes evolving in opposite directions within a unified thermodynamic description.
Paper Structure (16 sections, 1 theorem, 38 equations, 1 figure)

This paper contains 16 sections, 1 theorem, 38 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Ensemble vs. inferential physics
  3. Inference in sensory transduction and metrology
  4. Foundations for a thermodynamic framework
  5. Identifying an integrating factor and a first law
  6. Other thermodynamic relationships, and a third law
  7. Information gain and efficiency
  8. Information gain
  9. Information efficiency
  10. Discussion
  11. Appendix 1: Sensory background and adaptation results
  12. Linearization of sampling dynamics
  13. A Tweedie scaling ansatz and an expression for entropy
  14. Sensory adaptation
  15. A universal inequality for sensory adaptation
  16. ...and 1 more sections

Key Result

Theorem 1

Let $\mu(t)$ be a cyclic stimulus that generates a closed stimulus--response trajectory in $(\mu,m)$ via the state space model described in Section sec:inference. The change in information over a cycle obeys the second-law--type inequality

Figures (1)

  • Figure 1: Figure reproduced from wong2023fundamental. Steady-state activity plotted versus peak activity for a number of auditory studies. In all panels, the dashed lines show the theoretical upper and lower bounds of (\ref{['inequality']}). No fitted parameters were required to plot these bounds. SS vs PR from (a-g) single or averaged guinea pig fibre recordings (figs. 11a-d, 12, 17a and 4a from smith1975short); (h) saccular nerve fibres of goldfish (fig. 3 from fay1978coding); (i-m) single fibre gerbil recordings (figs. 4 and 5 from westerman1984rapid); (n-q) single guinea pig fibre recordings (figs. 1 and 2 from yates1985very); (r-s) single guinea pig fibre recordings (figs 3a and 3b from muller1991relationship); (t) averaged ferret data (fig. 6 from sumner2012auditory); (u-x) single cat fibre recordings (figs. 12e-h from peterson2021simplified). The spontaneous activity of each unit is indicated by an arrow pointing towards the x-axis.

Theorems & Definitions (2)

  • Theorem : Information inequality
  • proof