Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Phase Transitions in the Output Distribution of Large Language Models

Julian Arnold, Flemming Holtorf, Frank Schäfer, Niels Lörch

TL;DR

This work introduces a physics-inspired, distribution-based framework for automatically detecting phase-transition-like behavior in the output of large language models. It formalizes transitions as rapid changes in the conditional distribution $P(\cdot|T)$ and quantifies them with symmetric $f$-divergences (notably $D_{\mathrm{TV}}$ and $D_{\mathrm{JS}}$), linking small parameter shifts to Fisher information $\\mathcal{F}(T)$. A practical, low-variance signal, the linear dissimilarity with $g(x)=2x-1$, is developed and implemented by comparing left/right segments of the control-parameter grid, enabling efficient, black-box analysis of prompts, temperature, and training epochs. Applying the method to Pythia, Mistral, and Llama models reveals distinct transitions, including prompt-induced, tokenizer-boundary, and temperature-driven phase changes, and shows how transitions co-occur with rapid weight-distribution shifts during training. The approach promises scalable discovery of new behavioral phases in rapidly evolving LLMs and offers a principled tool for understanding and guiding model development and deployment.

Abstract

In a physical system, changing parameters such as temperature can induce a phase transition: an abrupt change from one state of matter to another. Analogous phenomena have recently been observed in large language models. Typically, the task of identifying phase transitions requires human analysis and some prior understanding of the system to narrow down which low-dimensional properties to monitor and analyze. Statistical methods for the automated detection of phase transitions from data have recently been proposed within the physics community. These methods are largely system agnostic and, as shown here, can be adapted to study the behavior of large language models. In particular, we quantify distributional changes in the generated output via statistical distances, which can be efficiently estimated with access to the probability distribution over next-tokens. This versatile approach is capable of discovering new phases of behavior and unexplored transitions -- an ability that is particularly exciting in light of the rapid development of language models and their emergent capabilities.

Phase Transitions in the Output Distribution of Large Language Models

TL;DR

This work introduces a physics-inspired, distribution-based framework for automatically detecting phase-transition-like behavior in the output of large language models. It formalizes transitions as rapid changes in the conditional distribution and quantifies them with symmetric -divergences (notably and ), linking small parameter shifts to Fisher information . A practical, low-variance signal, the linear dissimilarity with , is developed and implemented by comparing left/right segments of the control-parameter grid, enabling efficient, black-box analysis of prompts, temperature, and training epochs. Applying the method to Pythia, Mistral, and Llama models reveals distinct transitions, including prompt-induced, tokenizer-boundary, and temperature-driven phase changes, and shows how transitions co-occur with rapid weight-distribution shifts during training. The approach promises scalable discovery of new behavioral phases in rapidly evolving LLMs and offers a principled tool for understanding and guiding model development and deployment.

Abstract

In a physical system, changing parameters such as temperature can induce a phase transition: an abrupt change from one state of matter to another. Analogous phenomena have recently been observed in large language models. Typically, the task of identifying phase transitions requires human analysis and some prior understanding of the system to narrow down which low-dimensional properties to monitor and analyze. Statistical methods for the automated detection of phase transitions from data have recently been proposed within the physics community. These methods are largely system agnostic and, as shown here, can be adapted to study the behavior of large language models. In particular, we quantify distributional changes in the generated output via statistical distances, which can be efficiently estimated with access to the probability distribution over next-tokens. This versatile approach is capable of discovering new phases of behavior and unexplored transitions -- an ability that is particularly exciting in light of the rapid development of language models and their emergent capabilities.
Paper Structure (30 sections, 20 equations, 4 figures)

This paper contains 30 sections, 20 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Specific Findings.
  3. Methodology
  4. Quantifying Dissimilarity between Distributions
  5. Detecting Phase Transitions
  6. Application to Language Models and Numerical Implementation
  7. Utilized Large Language Models
  8. Pythia
  9. Mistral
  10. Llama
  11. Results
  12. Transitions as a Function of a Variable in the Prompt
  13. Transitions as a Function of the Model's Temperature
  14. Transitions as a Function of the Training Epoch
  15. Related Works
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Mistral model applied to the integer ordering prompt. (a) Different $g$-dissimilarities with $L=3$. (b) Linear dissimilarity for different $L$-values. [Number of text outputs generated per parameter value $T$: 10280. Number of generated output tokens: 10. Error bars indicate standard error of the mean over 4 batches, each with batch size 2056.]
  • Figure 2: Benchmarking of various models using the linear dissimilarity with $L=3$. (a) Test of ability to compare integers in value. (b) Bare integers as prompt reveals transition in tokenizer encoding. [Same numerical settings as in Fig \ref{['fig:prompt_token_mistral']}.]
  • Figure 3: Temperature transitions of Pythia 70M model in response to the prompt "There’s measuring the drapes, and then there’s measuring the drapes on a house you haven’t bought, a" -- an excerpt from OpenWebText gokaslan:2019. Linear dissimilarity measure ($L=5$) is shown in black. Heat capacity is shown in red. Dashed lines indicate local maxima, i.e., predicted critical points. Shaded regions indicate the error bands. (a) Temperature range $[10^{-4}, 2]$. (b) Zoomed-in range $[10^{-4}, 0.2]$ near $T=0$. [Number of text outputs generated per parameter value $T$: 20480. Number of generated output tokens: 10. Error bars indicate standard error of the mean over 4 batches, each with batch size 5120.]
  • Figure 4: Linear dissimilarity by epoch, with checkpoints taken every 1000 epochs. (a) Computed at $L=6$ for both weights and responses to 20 random prompts from OpenWebText (gray) and 7 short prompts (black) shown in panel (b). (b) Computed at $L=1$ for several prompts. For reference, the mean linear dissimilarity over short prompts and OpenWebText prompts with $L=1$ is also shown. [Number of text outputs generated per parameter value $T$ and prompt: 1536. Number of generated output tokens: 10. Error bars indicate the standard error of the mean over all corresponding prompts. Error bars for the individual prompts in panel (b) are almost negligible and thus omitted to avoid visual clutter.]