Deep Self-Evolving Reasoning

TL;DR

DSER reframes iterative verification and refinement as a self-evolving stochastic process and analyzes it through a Markov-chain lens, proving convergence to correct solutions under a modest bias toward improvement. When applied to an 8B-parameter reasoning model on AIME 2024-2025, DSER enables majority voting to solve several problems previously deemed unsolvable and even surpasses a 600B teacher in some settings, demonstrating a favorable trade-off between test-time computation and model scale. The framework highlights fundamental limitations of open-weight reasoners in self-verification and self-correction, while outlining concrete directions for reinforcement learning and exploration strategies to improve self-evolution dynamics. Overall, DSER offers a scalable, verification-agnostic paradigm for extending reasoning horizons in smaller models and provides a clear research agenda for developing intrinsically self-evolving capabilities.

Abstract

Long-form chain-of-thought reasoning has become a cornerstone of advanced reasoning in large language models. While recent verification-refinement frameworks have enabled proprietary models to solve Olympiad-level problems, their effectiveness hinges on strong, reliable verification and correction capabilities, which remain fragile in open-weight, smaller-scale models. This work demonstrates that even with weak verification and refinement capabilities on hard tasks, the reasoning limits of such models can be substantially extended through a probabilistic paradigm we call Deep Self-Evolving Reasoning (DSER). We conceptualize iterative reasoning as a Markov chain, where each step represents a stochastic transition in the solution space. The key insight is that convergence to a correct solution is guaranteed as long as the probability of improvement marginally exceeds that of degradation. By running multiple long-horizon, self-evolving processes in parallel, DSER amplifies these small positive tendencies, enabling the model to asymptotically approach correct answers. Empirically, we apply DSER to the DeepSeek-R1-0528-Qwen3-8B model. On the challenging AIME 2024-2025 benchmark, DSER solves 5 out of 9 previously unsolvable problems and boosts overall performance, enabling this compact model to surpass the single-turn accuracy of its 600B-parameter teacher through majority voting. Beyond its immediate utility for test-time scaling, the DSER framework serves to diagnose the fundamental limitations of current open-weight reasoners. By clearly delineating their shortcomings in self-verification, refinement, and stability, our findings establish a clear research agenda for developing next-generation models with powerful, intrinsic self-evolving capabilities.

Figures (7)

• Figure 1: Deep self-evolving reasoning enables DeepSeek-R1-0528-Qwen3-8B to solve 5 of 9 AIME 2024-2025 problems previously deemed "unsolvable" by standard majority voting over parallel trials (Avg@64: average accuracy over $64$ runs, Cons@64: consistency accuracy over $64$ runs). A notable example is the success for a difficult problem in AIME 2025 after $80$ self-evolving iterations, a process consuming approximately 10 million reasoning tokens. The final correct answer can be determined by a majority vote across the last ten self-evolving iterations.
• Figure 2: An overview of our DSER approach, where each rectangle of "Solve", "Verify", and "Refine" corresponds to one LLM reasoning call. In the view of Markov chain, a sufficient condition to elicit correct solutions for hard problems is to self-evolve deeply.
• Figure 3: Through the lens of Markov chain, we revisit the iterative verification-refinement cycle proposed by huang2025IMO-Gold-Agent. Here we need to define multiple states of solution correctness indexed by the number of consecutive self-verified rejections and refinements. For instance, $C^{(9)}$ denotes the solution being correct after $9$ consecutive rounds of self-verified rejections and refinements.
• Figure 4: Overall performance of DS-8B with DSER over iterations on the full AIME 2024 and 2025 benchmarks. We specifically flag the Avg@16 metric reported for DeepSeek-R1-0528, which is the 600B distillation teacher model for DS-8B.
• Figure 5: Per-question performance improvements on hard problems over self-evolving iterations, highlighting the diverse convergence speeds and stationary distributions of solution correctness.