Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformers

Krishnakumar Balasubramanian, Sayan Banerjee, Philippe Rigollet

TL;DR

This work develops a Fourier‑analytic framework for stationary McKean–Vlasov equations on the circle, transforming the problem into an infinite‑dimensional quadratic system in Fourier coefficients. It delivers a precise local bifurcation theory via Crandall–Rabinowitz, including multi‑mode and resonance phenomena, and connects these to discontinuous phase transitions in the free energy. The authors apply the theory to Noisy Mean‑Field Transformer models, revealing how the temperature (noise) parameter $eta$ shapes an intricate bifurcation landscape, metastability, and phase‑transition behavior, including clustering of modes at large $eta$ and explicit two‑term density approximations. They also derive global structures, such as Poisson‑kernel and Kuramoto‑type global branches, and establish comprehensive results on the global minimizers of the free energy, providing a coherent global portrait of the energy landscape and transitions between homogeneous and multimodal states.

Abstract

We study stationary solutions of McKean-Vlasov equations on the circle. Our main contributions stem from observing an exact equivalence between solutions of the stationary McKean-Vlasov equation and an infinite-dimensional quadratic system of equations over Fourier coefficients, which allows explicit characterization of the stationary states in a sequence space rather than a function space. This framework provides a transparent description of local bifurcations, characterizing their periodicity, and resonance structures, while accommodating singular potentials. We derive analytic expressions that characterize the emergence, form and shape (supercritical, critical, subcritical or transcritical) of bifurcations involving possibly multiple Fourier modes and connect them with discontinuous phase transitions. We also characterize, under suitable assumptions, the detailed structure of the stationary bifurcating solutions that are accurate upto an arbitrary number of Fourier modes. At the global level, we establish regularity and concavity properties of the free energy landscape, proving existence, compactness, and coexistence of globally minimizing stationary measures, further identifying discontinuous phase transitions with points of non-differentiability of the minimum free energy map. As an application, we specialize the theory to the Noisy Mean-Field Transformer model, where we show how changing the inverse temperature parameter $β$ affects the geometry of the infinitely many bifurcations from the uniform measure. We also explain how increasing $β$ can lead to a rich class of approximate multi-mode stationary solutions which can be seen as `metastable states'. Further, a sharp transition from continuous to discontinuous (first-order) phase behavior is observed as $β$ increases.

On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformers

TL;DR

This work develops a Fourier‑analytic framework for stationary McKean–Vlasov equations on the circle, transforming the problem into an infinite‑dimensional quadratic system in Fourier coefficients. It delivers a precise local bifurcation theory via Crandall–Rabinowitz, including multi‑mode and resonance phenomena, and connects these to discontinuous phase transitions in the free energy. The authors apply the theory to Noisy Mean‑Field Transformer models, revealing how the temperature (noise) parameter shapes an intricate bifurcation landscape, metastability, and phase‑transition behavior, including clustering of modes at large and explicit two‑term density approximations. They also derive global structures, such as Poisson‑kernel and Kuramoto‑type global branches, and establish comprehensive results on the global minimizers of the free energy, providing a coherent global portrait of the energy landscape and transitions between homogeneous and multimodal states.

Abstract

We study stationary solutions of McKean-Vlasov equations on the circle. Our main contributions stem from observing an exact equivalence between solutions of the stationary McKean-Vlasov equation and an infinite-dimensional quadratic system of equations over Fourier coefficients, which allows explicit characterization of the stationary states in a sequence space rather than a function space. This framework provides a transparent description of local bifurcations, characterizing their periodicity, and resonance structures, while accommodating singular potentials. We derive analytic expressions that characterize the emergence, form and shape (supercritical, critical, subcritical or transcritical) of bifurcations involving possibly multiple Fourier modes and connect them with discontinuous phase transitions. We also characterize, under suitable assumptions, the detailed structure of the stationary bifurcating solutions that are accurate upto an arbitrary number of Fourier modes. At the global level, we establish regularity and concavity properties of the free energy landscape, proving existence, compactness, and coexistence of globally minimizing stationary measures, further identifying discontinuous phase transitions with points of non-differentiability of the minimum free energy map. As an application, we specialize the theory to the Noisy Mean-Field Transformer model, where we show how changing the inverse temperature parameter affects the geometry of the infinitely many bifurcations from the uniform measure. We also explain how increasing can lead to a rich class of approximate multi-mode stationary solutions which can be seen as `metastable states'. Further, a sharp transition from continuous to discontinuous (first-order) phase behavior is observed as increases.
Paper Structure (33 sections, 18 theorems, 219 equations, 5 figures)

This paper contains 33 sections, 18 theorems, 219 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Background and challenges
  3. Our contributions
  4. General results on stationary McKean-Vlasov equations on $\mathbb{S}^1$
  5. Applications to noisy mean-field transformers
  6. Additional related works
  7. Solution spaces
  8. Local properties
  9. A Fourier representation
  10. First order bifurcations
  11. Specialization to noisy mean-field Transformers
  12. Periodicity
  13. Higher order bifurcations
  14. Quantifying clustering and metastability in Noisy Transformer for large $\beta$
  15. Stationary density representation in the supercritical bifurcation regime
  16. ...and 18 more sections

Key Result

Theorem 2.1

Fix $\kappa>0$. Let $p$ be any solution to eq:stat in $H_s^1(\mathbb{S}^1)$. Write Then, $\{ p_\ell: \ell \geq 0\}$ satisfy $p_0=1/2$ and for $\ell\geq 1$, Conversely, any $p \in H_s^1(\mathbb{S}^1)$ whose Fourier coefficients satisfy the above is a solution to eq:stat. Moreover, any such solution satisfies the fixed point equation where $Z = \int_0^{2\pi}\exp\left(\kappa \sum_{\ell=1}^\infty p

Figures (5)

  • Figure 1: Local pitchfork (i.e., subcritical and supercritical) and transcritical bifurcation types for the McKean-Vlasov stationary equation near the trivial solution at $(p,\kappa)=(0,\kappa^*)$: Solid (blue) line corresponds to the subcritical case. Dashed (red) line corresponds to the supercritical case. Dashed and dotted (dark green) line corresponds to the transcritical case. The vertical axis is the dimensionless deviation $\kappa-\kappa^*$ (zero at the bifurcation point) and the horizontal axis is the modal amplitude $s \;=\; \langle e_{\ell^*},\, p\rangle$. The vertical axis corresponds to the uniform solution $p_0$.
  • Figure 2: Plot of $R_\ell(\beta)$ vs. $\beta$: For fixed $\ell$, $R$ is strictly decreasing in $\beta$ and for every $\beta$, $R$ is increasing in $\ell$.
  • Figure 3: Visualization of the density in \ref{['eq:densitybifur']} for a few representative small and large $\beta$ and $\kappa$ values, for $m=1$, computed numerically by truncating the infinite series at 10. Solid and dashed densities correspond to $p^+_\kappa$ and $p^{-}_\kappa$ respectively.
  • Figure 4: The dashed (red) curve represents the free energy minimizing branch. The solid (blue) curve represents bifurcation branch with $\kappa < \kappa^*$.
  • Figure 5: Visualization of the Poisson kernel in \ref{['eq:PS']} for various wave numbers.

Theorems & Definitions (46)

  • Theorem 2.1
  • Lemma 2.1
  • Definition 2.1
  • Theorem 2.2
  • Remark 2.1
  • Theorem 2.3
  • Remark 2.2
  • Remark 2.3
  • Theorem 2.4
  • Remark 2.4
  • ...and 36 more