iReasoner: Trajectory-Aware Intrinsic Reasoning Supervision for Self-Evolving Large Multimodal Models

TL;DR

iReasoner introduces a trajectory-aware intrinsic CoT agreement reward to train self-evolving large multimodal models in a fully unsupervised setting. By aligning intermediate reasoning traces within the dominant-answer group and integrating this signal with traditional answer-level self-consistency, it directly supervises CoT steps without ground-truth labels. Experiments on unlabeled images show consistent gains across eight multimodal benchmarks, with notable improvements on tasks where intermediate reasoning is informative and transferable. The work demonstrates a practical path toward reasoning-aware self-improvement and provides insights into the design of step-level supervision in self-evolving multimodal systems.

Abstract

Recent work shows that large multimodal models (LMMs) can self-improve from unlabeled data via self-play and intrinsic feedback. Yet existing self-evolving frameworks mainly reward final outcomes, leaving intermediate reasoning weakly constrained despite its importance for visually grounded decision making. We propose iReasoner, a self-evolving framework that improves an LMM's implicit reasoning by explicitly eliciting chain-of-thought (CoT) and rewarding its internal agreement. In a Proposer--Solver loop over unlabeled images, iReasoner augments outcome-level intrinsic rewards with a trajectory-aware signal defined over intermediate reasoning steps, providing learning signals that distinguish reasoning paths leading to the same answer without ground-truth labels or external judges. Starting from Qwen2.5-VL-7B, iReasoner yields up to $+2.1$ points across diverse multimodal reasoning benchmarks under fully unsupervised post-training. We hope this work serves as a starting point for reasoning-aware self-improvement in LMMs in purely unsupervised settings.

Figures (8)

• Figure 1: iReasoner's intrinsic step-level CoT agreement. Given an unlabeled image, the Proposer generates a visually grounded question, and the Solver samples $N$ reasoning rollouts, each producing a CoT with multiple intermediate steps and a final answer (3 rollouts and 3 steps are shown here). Among rollouts in the dominant (majority-answer) group, we embed each step text $s_{i,j}$ into $e_{i,j}$ and form a per-step prototype $\mu_j$. Step agreement is computed via similarity $\mathrm{sim}(e_{i,j},\mu_j)$ and aggregated with higher weight on earlier, grounding-heavy steps ($w_1>w_2>w_3$) to produce a scalar Intrinsic CoT Agreement Reward.
• Figure 2: Overview of our iReasoner pipeline. From an unlabeled image $x$, a Proposer policy $\pi_p$ generates a question $q$, and a Solver policy $\pi_s$ produces $N$ sampled reasoning rollouts, inducing an empirical answer distribution $p(a\mid x,q)$. The distribution entropy provides an uncertainty-shaped reward for the Proposer and selects the dominant answer group for Solver-side step supervision. The Solver reward combines answer-level self-consistency with an intrinsic step-level agreement signal computed over intermediate reasoning traces by (i) extracting numbered steps, (ii) embedding each step, and (iii) forming per-step prototypes within the dominant-answer group. This yields intrinsic step-level supervision without annotated question--answer pairs or external verifiers.
• Figure 3: Question shaping and Solver reasoning behavior over training steps.(Top)Proposer reward remains stable (around $0.3$--$0.5$) while answer entropy stays in a moderate band (roughly $0.6$--$1.1$ nats), consistent with sustained intermediate difficulty rather than degeneracy. (Bottom) Majority-group density and mean step similarity increase over training, indicating a larger fraction of Solver samples agree on the dominant answer and that their intermediate steps become more aligned.
• Figure 3: Sensitivity to the maximum number of reasoning steps. We report performance across all eight benchmarks while varying the maximum number of parsed reasoning steps used to extract intermediate reasoning structure. Smaller step budgets truncate useful intermediate information, while overly large budgets can introduce noisy or redundant steps. The default setting of 8 steps used in our main experiments provides a strong balance and achieves robust performance across both general visual understanding and visual mathematics tasks.
• Figure 4: Outcome-only self-consistency treats distinct CoTs similarly. For the same image-question pair, three Solver rollouts produce the same final answer, but their intermediate steps differ: Rollouts 1 and 3 follow a consistent signed-area decomposition, while Rollout 2 deviates via incorrect intermediate claims yet still ends at the same answer. Since outcome-only intrinsic rewards depend only on answer agreement, these rollouts receive nearly identical learning signal despite qualitatively different reasoning traces, motivating step-aware supervision in iReasoner.