The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective

TL;DR

This work tackles why zeroth-order ES can fine-tune billion-parameter LLMs with small populations and why stochastic fine-tuning displays rise-then-decay under fixed noise. It introduces a variance-curvature framework in which a small set of high-curvature directions dominates improvement, and uses ES as a geometric probe with Gaussian smoothing to reveal coarse-grained landscape structure. The authors provide a tractable toy model showing how heterogeneous curvature and fixed stochasticity yield non-monotonic trajectories, and they empirically validate that improving perturbations remain accessible across model scales (0.5B–7B) with modest $N$ (around 30–40). They also formalize concepts of accessibility and degeneracy, demonstrating that population requirements do not scale with model size and that a curvature-active subspace governs improvement, suggesting broader algorithmic opportunities beyond the traditional curse-of-dimensionality mindset. Overall, the work reframes fine-tuning as a low-dimensional, geometry-driven process where diversity of improving perturbations can sustain scalable, stable optimization in high dimensions.

Abstract

Weight-perturbation evolution strategies (ES) can fine-tune billion-parameter language models with surprisingly small populations (e.g., $N\!\approx\!30$), contradicting classical zeroth-order curse-of-dimensionality intuition. We also observe a second seemingly separate phenomenon: under fixed hyperparameters, the stochastic fine-tuning reward often rises, peaks, and then degrades in both ES and GRPO. We argue that both effects reflect a shared geometric property of fine-tuning landscapes: they are low-dimensional in curvature. A small set of high-curvature dimensions dominates improvement, producing (i) heterogeneous time scales that yield rise-then-decay under fixed stochasticity, as captured by a minimal quadratic stochastic-ascent model, and (ii) degenerate improving updates, where many random perturbations share similar components along these directions. Using ES as a geometric probe on fine-tuning reward landscapes of GSM8K, ARC-C, and WinoGrande across Qwen2.5-Instruct models (0.5B--7B), we show that reward-improving perturbations remain empirically accessible with small populations across scales. Together, these results reconcile ES scalability with non-monotonic training dynamics and suggest that high-dimensional fine-tuning may admit a broader class of viable optimization methods than worst-case theory implies.

Figures (17)

• Figure 1: Non-monotonic training reward showing reward degradation beyond a peak training iteration $t_{\rm{peak}}$ in ES and GRPO fine-tuning of Qwen2.5-1.5B-Instruct on GSM8K gsm8k. Since GRPO does not directly maximize the task reward, training rewards plotted here are evaluated over the training set at every iteration. Both methods are trained on the same training set of 100 samples.
• Figure 2: Water-filling schematic for rise--then--decay dynamics.(a) Early: fast improvement along stiff directions while variance is small. (b) Peak: stiff directions are mostly exhausted; variance has risen enough to limit gains. (c) Late: variance-dominated drift along weakly constrained directions yields degradation under fixed stochasticity.
• Figure 3: Quadratic toy model: time-scale separation produces rise--then--decay. ES on a quadratic landscape with a two-block spectrum ($D{=}128$, $d{=}16$, $\lambda_h{=}1.0$, $\lambda_l{=}0.05$). Solid curves: Monte Carlo ES runs for different populations $N$ (noise levels). Dashed curves: closed-form prediction from the quadratic stochastic model (Appendix \ref{['app:toy_derivation']}). Larger $N$ (smaller $\kappa=\sigma^2/N$) raises the terminal plateau and suppresses late-time degradation.
• Figure 4: Schematic curvature structure: near-zero bulk with a few stiff outliers. The near-zero bulk corresponds to many weakly constrained directions, while a small number of outliers correspond to curvature-dominant directions that relax quickly and drive early improvement.
• Figure 5: Needle-in-a-haystack versus degenerate "wheel of fortune."(a) In a classical curse-of-dimensionality intuition, improvement is confined to a single unique direction with vanishing probability mass, so fixed-population random search fails as dimension grows. (b) Under low-dimensional curvature, improvement is governed by a small curvature-active subspace, but many ambient perturbations share similar projections onto this subspace (degeneracy), yielding an improvement-supporting set with nontrivial probability mass. Extreme-value selection (best-of-$N$) can therefore succeed with a small, fixed population size.