Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using physics-inspired Singular Learning Theory to understand grokking & other phase transitions in modern neural networks

Anish Lakkapragada

TL;DR

This paper argues that singularity in neural networks undermines classical statistical guarantees and proposes Singular Learning Theory (SLT) as a physics-inspired framework to study learning dynamics via the free energy $F_n$ and local learning coefficients $\lambda_\alpha$. It empirically tests SLT in toy settings, examining an Arrhenius-style rate hypothesis for grokking and phase transitions, and explores how LLC scales with problem difficulty across polynomial regressors, low-rank matrix factorization, and autoencoders. Some experiments align with SLT predictions (e.g., LLC scaling in low-rank factorization; linear LLC in bottleneck autoencoders), while others yield mixed or inconclusive results, highlighting both the strengths and limits of SLT in practical neural settings. The work establishes SLT as a promising tool for understanding neural phase transitions and outlines concrete open questions for future research, including methods to estimate free-energy barriers and the impact of domain constraints on singularities.

Abstract

Classical statistical inference and learning theory often fail to explain the success of modern neural networks. A key reason is that these models are non-identifiable (singular), violating core assumptions behind PAC bounds and asymptotic normality. Singular learning theory (SLT), a physics-inspired framework grounded in algebraic geometry, has gained popularity for its ability to close this theory-practice gap. In this paper, we empirically study SLT in toy settings relevant to interpretability and phase transitions. First, we understand the SLT free energy $\mathcal{F}_n$ by testing an Arrhenius-style rate hypothesis using both a grokking modulo-arithmetic model and Anthropic's Toy Models of Superposition. Second, we understand the local learning coefficient $λ_α$ by measuring how it scales with problem difficulty across several controlled network families (polynomial regressors, low-rank linear networks, and low-rank autoencoders). Our experiments recover known scaling laws while others yield meaningful deviations from theoretical expectations. Overall, our paper illustrates the many merits of SLT for understanding neural network phase transitions, and poses open research questions for the field.

Using physics-inspired Singular Learning Theory to understand grokking & other phase transitions in modern neural networks

TL;DR

This paper argues that singularity in neural networks undermines classical statistical guarantees and proposes Singular Learning Theory (SLT) as a physics-inspired framework to study learning dynamics via the free energy and local learning coefficients . It empirically tests SLT in toy settings, examining an Arrhenius-style rate hypothesis for grokking and phase transitions, and explores how LLC scales with problem difficulty across polynomial regressors, low-rank matrix factorization, and autoencoders. Some experiments align with SLT predictions (e.g., LLC scaling in low-rank factorization; linear LLC in bottleneck autoencoders), while others yield mixed or inconclusive results, highlighting both the strengths and limits of SLT in practical neural settings. The work establishes SLT as a promising tool for understanding neural phase transitions and outlines concrete open questions for future research, including methods to estimate free-energy barriers and the impact of domain constraints on singularities.

Abstract

Classical statistical inference and learning theory often fail to explain the success of modern neural networks. A key reason is that these models are non-identifiable (singular), violating core assumptions behind PAC bounds and asymptotic normality. Singular learning theory (SLT), a physics-inspired framework grounded in algebraic geometry, has gained popularity for its ability to close this theory-practice gap. In this paper, we empirically study SLT in toy settings relevant to interpretability and phase transitions. First, we understand the SLT free energy by testing an Arrhenius-style rate hypothesis using both a grokking modulo-arithmetic model and Anthropic's Toy Models of Superposition. Second, we understand the local learning coefficient by measuring how it scales with problem difficulty across several controlled network families (polynomial regressors, low-rank linear networks, and low-rank autoencoders). Our experiments recover known scaling laws while others yield meaningful deviations from theoretical expectations. Overall, our paper illustrates the many merits of SLT for understanding neural network phase transitions, and poses open research questions for the field.

Paper Structure

This paper contains 14 sections, 3 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. LLC Estimation.
  4. Experiments
  5. Our Arrhenius Reaction Rate Hypothesis.
  6. Experiment One: Grokking on Modulo Arithmetic.
  7. Experiment Two: Phase Transitions on Anthropic's Toy Models of Superposition.
  8. Motivation.
  9. Experiment One: Polynomial Regressors of Increasing Degree.
  10. Experiment Two: Low-Rank Neural Network with Matrix Factorization.
  11. Experiment Three: Autoencoders of Increasing Bottleneck Dimension.
  12. Conclusion and Future Directions
  13. Future Directions.
  14. Appendix

Figures (7)

  • Figure 1: Diagram of the modulo arithmetic neural network we use for our experiments, taken from panickssery2023learningcoef. Note that this task can be thought of as taking two numbers in $\mathbb{Z}_p$ and outputting the modulo in $\mathbb{Z}_p$.
  • Figure 2: Distribution of $\Delta\lambda$ and $\log r_{i \to j}$.
  • Figure 3: $\Delta \mathcal{F}_{i \to j}$ vs. $\log r_{i \to j}$ with linear fit on modulo arithmetic toy networks with $p = 53$.
  • Figure 4: $\Delta \mathcal{F}_{i \to j}$ vs. $\log r_{i \to j}$ with linear fit on Anthropic's TMS models.
  • Figure 5: Estimated LLC versus degree of polynomial regressor for all choices of instance space $\mathcal{X}$.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Definition 3.1: Local Learning Coefficient (LLC)