No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning

TL;DR

The paper addresses critic staleness in critique-guided RL for open-world LLM agents by introducing ECHO, a synchronized co-evolution framework that jointly optimizes a policy and a critic through cascaded diagnostic rollouts and saturation-aware gains. ECHO uses a two-stage cascaded rollout to generate diverse critiques and refinements, coupled with a saturation-aware reward design and dual-track GRPO updates to keep feedback aligned with the evolving policy. Key contributions include identifying critic staleness under on-policy drift, proposing a synchronized co-evolutionary optimization mechanism, and demonstrating improved stability and long-horizon task success across multiple environments and backbones. The approach has practical impact for data-efficient, robust open-world agent learning, enabling critiques to remain useful as policies advance, though it relies on reliable reward signals and could benefit from unifying scoring and critique generation in future work.

Abstract

Critique-guided reinforcement learning (RL) has emerged as a powerful paradigm for training LLM agents by augmenting sparse outcome rewards with natural-language feedback. However, current methods often rely on static or offline critic models, which fail to adapt as the policy evolves. In on-policy RL, the agent's error patterns shift over time, causing stationary critics to become stale and providing feedback of diminishing utility. To address this, we introduce ECHO (Evolving Critic for Hindsight-Guided Optimization)}, a framework that jointly optimizes the policy and critic through a synchronized co-evolutionary loop. ECHO utilizes a cascaded rollout mechanism where the critic generates multiple diagnoses for an initial trajectory, followed by policy refinement to enable group-structured advantage estimation. We address the challenge of learning plateaus via a saturation-aware gain shaping objective, which rewards the critic for inducing incremental improvements in high-performing trajectories. By employing dual-track GRPO updates, ECHO ensures the critic's feedback stays synchronized with the evolving policy. Experimental results show that ECHO yields more stable training and higher long-horizon task success across open-world environments.

Figures (5)

• Figure 1: Comparison of critic paradigms. (a) Conventional Static Paradigms: Use decoupled, frozen critic modules initialized from off-the-shelf templates or fine-tuned separate models, resulting in static evaluation and inflexible feedback. (b) Our ECHO Paradigm: Policy and critic co-evolve organically. The policy first generates an initial rollout $\tau_o$, refined to $\tau_r$ using the critic’s diagnostic guidance $c$. Both models are jointly updated, ensuring the critic's diagnostic precision synchronizes with the policy's evolving failure patterns.
• Figure 2: Overview of ECHO training with saturation-aware (SA) critic rewards. At step $t$, the policy $\pi_{\theta_t}$ produces rollouts $\tau_o$, which are scored by a reward model to obtain $s_o$. A critic $\pi_{\psi_t}$ generates critiques that are appended to the original query to elicit refined rollouts $\tau_r$, scored as $s_r$. We compute the SA critic reward $r_c$ to emphasize last-mile improvements near saturation, and update the critic and policy synchronously to obtain $\pi_{\psi_{t+1}}$ and $\pi_{\theta_{t+1}}$.
• Figure 3: Failure-pattern drift across training phases. We visualize failed trajectories from early, intermediate, and late checkpoints in a diagnosis embedding space using t-SNE, with contours indicating density regions.
• Figure 4: Effect of saturation-aware gain shaping on last-mile refinement. We plot density scatter maps of pre-refinement and post-refinement rewards $(s_o, s_r)$ on WebShop and SciWorld using Qwen3-4B. Points in the green region satisfy $s_r > s_o$ and correspond to reward-improving refinements, where higher density indicates more effective critiques. The highlighted high-score square marks the near-ceiling regime.
• Figure 5: Training reward curves across four environments (Qwen3-4B).