ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learning

TL;DR

The paper tackles the instability of test-time reinforcement learning (TTRL) caused by high-entropy rollout branching and early pseudo-label bias. It introduces ECHO, a framework that jointly regulates entropy and confidence during tree-structured rollouts, uses online pruning to avoid entropy traps, and couples this with confidence-adaptive clipping and entropy–confidence hybrid advantage shaping during updates. The approach yields consistent gains across natural-language and multimodal mathematical reasoning benchmarks, and demonstrates robust generalization under constrained rollout budgets. By balancing exploration and stability without external supervision, ECHO offers a practical boost for reasoning-intensive inference with large models.

Abstract

Test-time reinforcement learning generates multiple candidate answers via repeated rollouts and performs online updates using pseudo-labels constructed by majority voting. To reduce overhead and improve exploration, prior work introduces tree structured rollouts, which share reasoning prefixes and branch at key nodes to improve sampling efficiency. However, this paradigm still faces two challenges: (1) high entropy branching can trigger rollout collapse, where the branching budget concentrates on a few trajectories with consecutive high-entropy segments, rapidly reducing the number of effective branches; (2) early pseudo-labels are noisy and biased, which can induce self-reinforcing overfitting, causing the policy to sharpen prematurely and suppress exploration. To address these issues, we propose Entropy Confidence Hybrid Group Relative Policy Optimization (ECHO). During rollout, ECHO jointly leverages local entropy and group level confidence to adaptively control branch width, and further introduces online confidence-based pruning to terminate persistently low confidence branches, avoiding high entropy traps and mitigating collapse. During policy updates, ECHO employs confidence adaptive clipping and an entropy confidence hybrid advantage shaping approach to enhance training robustness and mitigate early stage bias. Experiments demonstrate that ECHO achieves consistent gains on multiple mathematical and visual reasoning benchmarks, and generalizes more effectively under a limited rollout budget.

Figures (7)

• Figure 1: Performance overview of ECHO algorithm.
• Figure 2: Empirical study of high-entropy rollout collapse in entropy-driven tree search for TTRL.
• Figure 3: Overview of our proposed framework. Given a query, the LLM/VLM performs entropy–confidence guided tree-search rollouts with adaptive branching and confidence-based online pruning. The surviving trajectories produce candidate answers $y_i$, which are aggregated by majority voting to obtain $y'$. We then compute hybrid shaped advantages $A_i^{\mathrm{hyb}}$ for policy optimization, while pruning branches whose grouped confidence falls below $\tau_{\text{prune}}$ to save budget and prevent high-entropy collapse.
• Figure 4: Training dynamics for Ours and ETMR. Both models trained on AIME2024.
• Figure 5: Visualization of Rollout diversity: ETMR and Ours.

Theorems & Definitions (8)

• Lemma 3.1: Piecewise surrogate
• proof
• Lemma 3.2: Ratio gradient
• proof
• Proposition 3.3: Clipped policy gradient
• proof
• Lemma 3.4: KL gradient
• proof