SIGMA: Scalable Spectral Insights for LLM Collapse

TL;DR

The paper tackles model collapse from recursive synthetic-data training by introducing SIGMA, a spectral framework that tracks representation health through the Gram matrix spectrum of embeddings. It develops deterministic and stochastic bounds on the Gram determinant, enabling scalable collapse monitoring even when full eigendecomposition is intractable. Two practical diagnostics, Sigma-UB Track I and Track II, are derived to provide a conservative envelope and a sensitive trend probe, respectively, with a size-corrected baseline to distinguish genuine geometric contraction from dataset growth. Experiments in controlled settings reveal that carrying forward weights across generations accelerates collapse under recursion (S2) compared to restarting from base (S1), illustrating the practical value of SIGMA for monitoring and diagnosing recursive training pipelines. The framework offers a principled, scalable tool for ensuring healthy embedding geometry in large-scale LLMs and informs more robust data-generation and fine-tuning strategies.

Abstract

The rapid adoption of synthetic data for training Large Language Models (LLMs) has introduced the technical challenge of "model collapse"-a degenerative process where recursive training on model-generated content leads to a contraction of distributional variance and representational quality. While the phenomenology of collapse is increasingly evident, rigorous methods to quantify and predict its onset in high-dimensional spaces remain elusive. In this paper, we introduce SIGMA (Spectral Inequalities for Gram Matrix Analysis), a unified framework that benchmarks model collapse through the spectral lens of the embedding Gram matrix. By deriving and utilizing deterministic and stochastic bounds on the matrix's spectrum, SIGMA provides a mathematically grounded metric to track the contraction of the representation space. Crucially, our stochastic formulation enables scalable estimation of these bounds, making the framework applicable to large-scale foundation models where full eigendecomposition is intractable. We demonstrate that SIGMA effectively captures the transition towards degenerate states, offering both theoretical insights into the mechanics of collapse and a practical, scalable tool for monitoring the health of recursive training pipelines.

Figures (4)

• Figure 1: FIN, Setting S1 (restart-from-base): drift vs. generation under both SIGMA-UB tracks.Top: Track II $\Delta U_{\mathrm{LLN,cov}}(\delta)$. Bottom: Track I $\Delta G_{\mathrm{KF}}(\delta)$. Values are baseline drifts relative to the base checkpoint ($g=0$). S1 isolates data recursion only: synthetic text is regenerated each generation, but training restarts from the same base checkpoint. Appendix \ref{['app:plot_tables_fin']} reports the exact per-generation table used to render this figure.
• Figure 2: FIN, Setting S2 (true recursion): drift vs. generation under both SIGMA-UB tracks.Top: Track II $\Delta U_{\mathrm{LLN,cov}}(\delta)$. Bottom: Track I $\Delta G_{\mathrm{KF}}(\delta)$. Values are baseline drifts relative to the base checkpoint ($g=0$). S2 compounds data recursion + weight recursion: each generation regenerates synthetic text and continues training from the previous generation weights. Appendix \ref{['app:plot_tables_fin']} reports the exact per-generation table used to render this figure.
• Figure 3: Normalized overlay of Sigma-UB drifts and surface-form proxies (FIN). Each curve is min--max normalized to $[0,1]$ within its panel for visual comparability. We overlay four signals across generations: Track II drift $\Delta U_{\mathrm{LLN,cov}}(\delta)$, Track I drift $\Delta G_{\mathrm{KF}}(\delta)$, distinct-2, and hashed $n$-gram HHI. Left: S2 (true recursion). Right: S1 (restart-from-base). The key qualitative trend is that the large early Track-II contraction in S2 precedes (and is accompanied by) stronger surface-form concentration (rising HHI) than in S1.
• Figure 4: Bucket localization under S1 (FIN; optional).Sigma-UB drifts computed after restricting evaluation to each frozen prompt bucket. Bucket curves can be noisier than the aggregate because each bucket contains fewer prompts; we therefore interpret bucket localization as a qualitative "where does collapse concentrate?" tool rather than as a primary metric.

Theorems & Definitions (8)

• Theorem 1: Deterministic Spectral Bound
• Theorem 2: Stochastic Scaling Spectral Bound
• Theorem 3: Stochastic Error Term
• proof : Proof of Theorem \ref{['thm:deterministic']}
• proof : Proof of Theorem \ref{['thm:stochastic']}
• proof : Proof of Theorem \ref{['thm:err_stochastic']}
• Lemma 4: Convergence Rate of Deviation Matrix
• proof : Proof of Lemma \ref{['thm:rate_lemma']}