Theoretical Modeling of LLM Self-Improvement Training Dynamics Through Solver-Verifier Gap

TL;DR

The paper addresses how LLM self-improvement unfolds through a solver-verifier gap. It introduces a physics-inspired coupled-dynamics model with $\frac{dU_s}{dt}=-\alpha E(t)$ and $\frac{dU_v}{dt}=-\beta E(t)$, plus a linearized energy $E(t)\approx kG-b$ that yields exponential trajectories toward limits $U_s,\infty$ and $U_v,\infty$, with $U_s,\infty=\frac{1}{\alpha-\beta}(\alpha U_v,0-\beta U_s,0+\alpha \frac{b}{k})$ and $G_\infty=\frac{b}{k}$. Empirical results across multiple models and datasets validate the exponential dynamics, showing the verifier typically outperforms the solver and that the solver-verifier gap drives self-improvement. The work extends to cross-improvement with limited external data, deriving conditions under which external data boosts verification capability and final performance, and demonstrating allocation strategies that yield robust gains. Overall, the framework provides a quantitative understanding of self-improvement dynamics and a practical pathway to surpass inherent limits via cross-improvement.

Abstract

Self-improvement is among the most prominent techniques within the realm of large language models (LLM), aiming to enhance the LLM performance without relying on external data. Despite its significance, generally how LLM performances evolve during the self-improvement process remains underexplored. In this paper, we theoretically model the training dynamics of self-improvement via the concept of solver-verifier gap. This is inspired by the conjecture that the performance enhancement of self-improvement stems from the gap between LLM's solver capability and verifier capability. Based on the theoretical framework, we further show how to model the entire training trajectory. This framework allows quantifying the capability limit of self-improvement by fitting the theoretical model to the experiment results. We empirically validate the effectiveness of the theoretical framework on various LLMs and datasets. Beyond self-improvement, we extend our analysis to investigate how external data influences these dynamics within the framework. Notably, we find that under limited external data regimes, such external data can be utilized at any stage without significantly affecting final performances, which accords with the empirical observations.

Figures (18)

• Figure 1: Illustration of the theoretical results. The training dynamics are potentially empowered by the solver-verifier gap under the theoretical framework. For self-improvement (left), the accuracy exhibits exponential curves towards a limit. For cross-improvement (right), adding external data (a different verifier with limited queries) at different times yields similar performance.
• Figure 2: Verification on linear relationship between the Uncertainty gap $G$ and its rate of change $dG/dt$ on Phi-4-mini with QE method. The scatter points represent empirical data from self-improvement on the Math and GSM8k datasets, while the solid lines show the linear regression fits.
• Figure 3: Accuracy and uncertainty during the self-improvement of the Phi-4-mini model on the Math and GSM8k datasets using the QE method. The experimental results show that the accuracy increases during self-improvement process while the uncertainty decreases.
• Figure 4: Exponential trends of model uncertainty during self-improvement on train split. The results illustrate the uncertainty and gap of the solver and verifier for Phi-4-mini. The scatter points represent the measured data, while the solid lines are the best-fit curves to an exponential model. $R^2>0.9$ indicates that the evolution of these uncertainties is well-described by an exponential function.
• Figure 5: An overview of our theoretical framework.

Theorems & Definitions (4)

• Proposition 3.1
• Corollary 3.1
• Corollary 3.2
• Proposition 5.1