AERO: Autonomous Evolutionary Reasoning Optimization via Endogenous Dual-Loop Feedback

TL;DR

AERO tackles autonomous reasoning evolution in LLMs by embedding self-questioning, self-answering, and self-criticism within a dual-loop framework that eliminates external data or verifiers. It combines entropy-guided Zone of Proximal Development (ZPD) positioning with Independent Counterfactual Correction (ICC) to produce high-fidelity internal feedback, and employs a Staggered Training Strategy to stabilize curriculum progression. Policy updates are performed with Kahneman-Tversky optimization (KTO) on binary feedback, enabling offline, stable growth of multiple functional roles. Across nine benchmarks spanning mathematical, physical, and general reasoning, AERO yields consistent gains over data-free baselines and demonstrates robust, architecture-agnostic autonomous evolution.

Abstract

Large Language Models (LLMs) have achieved significant success in complex reasoning but remain bottlenecked by reliance on expert-annotated data and external verifiers. While existing self-evolution paradigms aim to bypass these constraints, they often fail to identify the optimal learning zone and risk reinforcing collective hallucinations and incorrect priors through flawed internal feedback. To address these challenges, we propose \underline{A}utonomous \underline{E}volutionary \underline{R}easoning \underline{O}ptimization (AERO), an unsupervised framework that achieves autonomous reasoning evolution by internalizing self-questioning, answering, and criticism within a synergistic dual-loop system. Inspired by the \textit{Zone of Proximal Development (ZPD)} theory, AERO utilizes entropy-based positioning to target the ``solvability gap'' and employs Independent Counterfactual Correction for robust verification. Furthermore, we introduce a Staggered Training Strategy to synchronize capability growth across functional roles and prevent curriculum collapse. Extensive evaluations across nine benchmarks spanning three domains demonstrate that AERO achieves average performance improvements of 4.57\% on Qwen3-4B-Base and 5.10\% on Qwen3-8B-Base, outperforming competitive baselines. Code is available at https://github.com/mira-ai-lab/AERO.

Figures (8)

• Figure 1: The AERO framework for unsupervised self-evolution. AERO internalizes three core capabilities, which include self-questioning, self-answering, and self-criticism, within a unified dual-loop system to enable autonomous growth without any reliance on external data or verifiers.
• Figure 2: The AERO framework consists of an inner loop for autonomous experience synthesis and an outer loop for preference-based policy optimization. Within the inner loop, the single model adopts generator, solver, and refiner roles to produce tasks and reasoning trajectories. These verified experiences are then utilized in the outer loop for policy update.
• Figure 3: Synchronous vs. Staggered Training Strategy. Colors indicate the progression of training rounds. Arrows illustrate the capability alignment between Self-Questioning (Q) and Self-Answering/Self-Criticism (A/C). While synchronous training leads to capability asynchrony, the staggered approach introduces a temporal offset to synchronize growth across all functional capabilities.
• Figure 4: Evolution of ZPD and response entropy across iterations. The rightward shift of the curves from Round 1 to Round 3 demonstrates the model's cognitive growth, as the ZPD dynamically advances toward higher difficulty levels.
• Figure 5: Normalized improvement trends relative to the base model across five training rounds. The results highlight the robust scalability of AERO across diverse model families (Llama-3.2, Qwen2.5) and parameter scales (3B to 32B) within the Mathematics, Physics, and General Reasoning domains.