Superposition unifies power-law training dynamics

TL;DR

The paper analyzes how feature superposition reshapes power-law training dynamics in a toy teacher–student model, showing that without superposition the training loss decays as $\mathcal{L}(t) \propto t^{-\alpha}$ with $\alpha=(a+2b-1)/a$, reflecting data and channel statistics. When a bottleneck induces superposition ($K<N$), the mid-training dynamics become universal with $\alpha \approx 1$, yielding substantial acceleration (up to ~10x) and independence from $a$ and $b$. The authors derive the no-superposition exponent analytically and confirm it empirically, then demonstrate universal acceleration under superposition across various $a,b,K$ and analyze optimal compute scaling and width-dependent final losses. They provide a mechanistic interpretation via feature mixing that equalizes effective gradients, offering insights for efficient training of large-scale models and informing future extensions to deeper or attention-based architectures.

Abstract

We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training without superposition, establishing that the power-law training exponent depends on both the input data statistics and channel importance. Remarkably, we discover that a superposition bottleneck induces a transition to a universal power-law exponent of $\sim 1$, independent of data and channel statistics. This one over time training with superposition represents an up to tenfold acceleration compared to the purely sequential learning that takes place in the absence of superposition. Our finding that superposition leads to rapid training with a data-independent power law exponent may have important implications for a wide range of neural networks that employ superposition, including production-scale large language models.

Figures (14)

• Figure 1: Geometric illustration of superposition.(a) In the absence of superposition, features have zero interference with each other. (b) Superposition compresses features into a smaller latent size, introducing interference among them (e.g., Feature 5 projects onto Feature 1).
• Figure 2: The teacher-student model setup under superposition, where $K \leq N$. The student learns in a compressed latent space via the embedding layers $\mathbf{W}$ and $\mathbf{W}^\top$. A bias and ReLU nonlinearity are applied to the output to suppress interference noise.
• Figure 3: Without superposition, learning exponents depend on both input data and channel statistics. We verify the analytic theory for the no-superposition baseline ($N=K=1024$). (a) Empirical loss curves (solid) track the theoretical predictions (dashed) across varying input decays $a$ and channel importances $b$. (b) The fitted power-law exponents align precisely with the derived scaling law $\alpha = (a+2b-1)/a$, confirming that learning is strictly governed by data and channel statistics.
• Figure 4: Mid-training acceleration via superposition. Loss from the superposition experiment ($N=1024$, $K = 512$) is compared to the no-superposition theory ($N=K=1024$). A mid-training acceleration in loss convergence appears under superposition despite the bottleneck in $K$.
• Figure 5: Universality of the training exponent under superposition. We plot the fitted power-law exponent $\alpha$ for varying student sizes $K \in \{128, 256, 512\}$. Unlike the sequential case where $\alpha$ varies with data and channels, superposition locks the exponent to $\boldsymbol{\alpha \approx 1}$ regardless of the input feature decay $a$ or the channel importance decay $b$.