Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evidence Slopes and Effective Dimension in Singular Linear Models

Kalyaan Rao

TL;DR

The paper investigates how Bayesian evidence deviates from Laplace/BIC in singular linear models, where the ambient parameter count $d$ over-penalises the marginal likelihood. Focusing on linear--Gaussian rank models and linear dictionaries, it derives closed-form exact evidences and shows the RLCT equals $\lambda(r)=r/2$, with the Laplace/BIC error scaling as $\frac{d-r}{2}\log n$. An RLCT-aware correction restores the correct evidence slope and is invariant to overcomplete reparameterisations that span the same subspace, while BIC is not. The authors further demonstrate that the slope of the exact evidence with respect to $\log n$ provides a practical estimator of the RLCT in these settings, offering a finite-sample handle on the effective dimension and highlighting the representation-invariance properties of RLCT-based penalties. These results give a concrete, tractable view of model selection in simple singular linear models and motivate extensions to broader non-linear settings.

Abstract

Bayesian model selection commonly relies on Laplace approximation or the Bayesian Information Criterion (BIC), which assume that the effective model dimension equals the number of parameters. Singular learning theory replaces this assumption with the real log canonical threshold (RLCT), an effective dimension that can be strictly smaller in overparameterized or rank-deficient models. We study linear-Gaussian rank models and linear subspace (dictionary) models in which the exact marginal likelihood is available in closed form and the RLCT is analytically tractable. In this setting, we show theoretically and empirically that the error of Laplace/BIC grows linearly with (d/2 minus lambda) times log n, where d is the ambient parameter dimension and lambda is the RLCT. An RLCT-aware correction recovers the correct evidence slope and is invariant to overcomplete reparameterizations that represent the same data subspace. Our results provide a concrete finite-sample characterization of Laplace failure in singular models and demonstrate that evidence slopes can be used as a practical estimator of effective dimension in simple linear settings.

Evidence Slopes and Effective Dimension in Singular Linear Models

TL;DR

The paper investigates how Bayesian evidence deviates from Laplace/BIC in singular linear models, where the ambient parameter count over-penalises the marginal likelihood. Focusing on linear--Gaussian rank models and linear dictionaries, it derives closed-form exact evidences and shows the RLCT equals , with the Laplace/BIC error scaling as . An RLCT-aware correction restores the correct evidence slope and is invariant to overcomplete reparameterisations that span the same subspace, while BIC is not. The authors further demonstrate that the slope of the exact evidence with respect to provides a practical estimator of the RLCT in these settings, offering a finite-sample handle on the effective dimension and highlighting the representation-invariance properties of RLCT-based penalties. These results give a concrete, tractable view of model selection in simple singular linear models and motivate extensions to broader non-linear settings.

Abstract

Bayesian model selection commonly relies on Laplace approximation or the Bayesian Information Criterion (BIC), which assume that the effective model dimension equals the number of parameters. Singular learning theory replaces this assumption with the real log canonical threshold (RLCT), an effective dimension that can be strictly smaller in overparameterized or rank-deficient models. We study linear-Gaussian rank models and linear subspace (dictionary) models in which the exact marginal likelihood is available in closed form and the RLCT is analytically tractable. In this setting, we show theoretically and empirically that the error of Laplace/BIC grows linearly with (d/2 minus lambda) times log n, where d is the ambient parameter dimension and lambda is the RLCT. An RLCT-aware correction recovers the correct evidence slope and is invariant to overcomplete reparameterizations that represent the same data subspace. Our results provide a concrete finite-sample characterization of Laplace failure in singular models and demonstrate that evidence slopes can be used as a practical estimator of effective dimension in simple linear settings.
Paper Structure (25 sections, 4 theorems, 31 equations, 4 figures, 2 tables)

This paper contains 25 sections, 4 theorems, 31 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Laplace approximation and BIC.
  4. Singular learning theory.
  5. Empirical flatness and effective dimension.
  6. Low-rank and overparameterised models.
  7. Background and Problem Setup
  8. Bayesian evidence and Laplace approximation
  9. Singular learning theory and RLCT
  10. Linear--Gaussian rank models and dictionaries
  11. Rank-$r$ linear regression.
  12. Linear subspace (dictionary) model.
  13. Exact and approximate evidences.
  14. Evidence slopes in rank-$r$ linear regression
  15. Closed-form marginal likelihood
  16. ...and 10 more sections

Key Result

Proposition 1

Let $y \in \mathbb{R}^n$ and $A_n\in\mathbb{R}^{n\times d}$ be fixed, and consider Then the marginal likelihood $Z_n = p(y \mid A_n)$ is where $S_n := A_n^\top A_n$ and $\alpha := \tau^2/\sigma^2$.

Figures (4)

  • Figure 1: Rank sweep in linear regression. For each intrinsic rank $r$ we estimate the slope of the BIC error $\Delta_{\text{BIC}}(n)$ and the RLCT-corrected error $\Delta_{\text{RLCT}}(n)$ versus $\log n$. As $r$ approaches the ambient dimension $d$, the BIC slope approaches zero. For smaller $r$ the BIC slope is strongly negative, while the RLCT-corrected slope stays near zero across all $r$.
  • Figure 2: Regular model: $\Delta_{\text{BIC}}(n)$ and $\Delta_{\text{RLCT}}(n)$ versus $\log n$. Both slopes are close to zero, as predicted when $\lambda=d/2$.
  • Figure 3: Singular model: $\Delta_{\text{BIC}}(n)$ and $\Delta_{\text{RLCT}}(n)$ versus $\log n$. The BIC error has a large negative slope, while the RLCT-corrected error stays nearly flat.
  • Figure 4: Eigenvalue spectra of $D^\top D$ (minimal dictionary, $d=r$) and $D'^\top D'$ (overcomplete dictionary, $d'>r$) for the same underlying subspace. Both have $r$ large eigenvalues and a block of near-zero eigenvalues, illustrating the redundant coordinates in the overcomplete representation.

Theorems & Definitions (9)

  • Proposition 1: Exact evidence in linear--Gaussian regression
  • proof
  • Proposition 2: RLCT for rank-$r$ regression
  • proof : Proof sketch
  • Proposition 3: Error of Laplace/BIC in rank-$r$ regression
  • proof
  • Proposition 4: Representation invariance of RLCT and evidence
  • proof : Proof sketch
  • Remark 1: Non-invariance of BIC in overcomplete dictionaries