Singular Fluctuation as Specific Heat in Bayesian Learning

Abstract

Singular learning theory characterizes Bayesian models with non-identifiable parameterizations through two central quantities: the real log canonical threshold (RLCT), which governs marginal likelihood asymptotics, and the singular fluctuation, which determines second-order generalization behavior and the complexity term in WAIC. While the geometric meaning of the RLCT is well understood, the interpretation of singular fluctuation has remained comparatively opaque. We show that singular fluctuation admits a precise thermodynamic interpretation. Under a tempered (Gibbs) posterior, it is exactly the curvature of the Bayesian free energy with respect to inverse temperature; equivalently, the variance of the log-likelihood observable. In this sense, singular fluctuation is the statistical analogue of specific heat. This identity clarifies why singular fluctuation controls the equation of state relating training and generalization error and explains the success of WAIC in singular models: WAIC estimates a fluctuation coefficient rather than a parameter dimension. Across Gaussian mixture models and reduced-rank regression, we demonstrate that singular fluctuation behaves as a thermodynamic response coefficient. As temperature decreases, posterior reorganization suppresses fluctuation directions that affect predictive performance, and model-specific geometric observables track the decay of singular fluctuation. Rather than introducing new asymptotic expansions, this work unifies existing variance identities, equation-of-state results, and WAIC complexity corrections under a single free-energy curvature framework.

Figures (4)

• Figure 1: Thermodynamic response in the two-component Gaussian mixture model. Left: Estimated singular fluctuation $\hat{\nu}(\beta)$ decreases smoothly with inverse temperature, indicating progressive suppression of posterior log-likelihood fluctuations. Right: Posterior occupancy variance $\mathrm{Var}_\beta(\rho)$ tracks $\hat{\nu}(\beta)$ closely across temperatures, demonstrating that singular fluctuation measures thermally active mixture assignment variability.
• Figure 2: Posterior occupancy distributions for the Gaussian mixture model at representative inverse temperatures. As $\beta$ increases, the distribution progressively concentrates, reflecting suppression of mixture assignment variability. This reorganization corresponds to the decay of $\hat{\nu}(\beta)$ shown in Figure \ref{['fig:gmm_combined']}.
• Figure 3: Thermodynamic response in reduced-rank regression. Left: Posterior variance of the Frobenius norm $\mathrm{Var}_\beta(\|B\|_F)$ (orange, left axis) plotted alongside the estimated singular fluctuation $\hat{\nu}(\beta)$ (blue, right axis). Both decrease smoothly with inverse temperature, indicating suppression of algebraic fluctuation directions under tempering. Right: Posterior variance of the effective rank $\mathrm{Var}_\beta(r_{\mathrm{eff}})$ (red, left axis) compared with $\hat{\nu}(\beta)$ (blue, right axis). Because $r_{\mathrm{eff}}$ is invariant under orthogonal reparameterization, its alignment with $\hat{\nu}(\beta)$ shows that singular fluctuation tracks intrinsic posterior geometric reorganization rather than coordinate artifacts.
• Figure 4: Empirical validation of the equation of state $E[G_n - T_n] \approx 2\nu/n$ averaged across independent datasets. Each blue point corresponds to a single independently generated dataset, with horizontal coordinate $2\hat{\nu}/n$ and vertical coordinate $\hat{G}_n - \hat{T}_n$. Agreement holds within Monte Carlo uncertainty. These diagnostics confirm consistency of the thermodynamic estimator but are not used as primary evidence for the specific-heat interpretation.

Theorems & Definitions (4)

• Proposition 3.1: Free-energy curvature identity
• proof
• Corollary 3.2: Singular fluctuation as specific heat at $\beta=1$
• proof