Co-Evolving Agents: Learning from Failures as Hard Negatives

TL;DR

<3-5 sentence high-level summary> The paper tackles the data-efficiency and generalization challenges of self-improving agents by introducing a co-evolving framework where a target agent learns alongside an auxiliary failure agent. The failure agent optimizes over failure trajectories to produce informative hard negatives, which are incorporated into the target's Direct Preference Optimization (DPO) with an auxiliary supervised loss for stability. Across WebShop, ScienceWorld, and InterCodeSQL, the approach yields consistent improvements over strong baselines, including ETO, and demonstrates notably better generalization to unseen tasks. The work shows that structured failure signals, learned through co-evolution, can serve as valuable learning signals for robust self-improvement in complex, real-world domains.

Abstract

The rapid progress of large foundation models has accelerated the development of task-specialized agents across diverse domains. However, the effectiveness of agents remains tightly coupled with the quality of training data, while curating task-specific datasets remains costly and often infeasible in real-world scenarios. Recent work has explored self-improving agents that autonomously generate, refine, and re-train on their own trajectories. A prominent line of approaches further leverages preference optimization by pairing predicted trajectories with scarce ground-truth trajectories, enabling agents to learn directly from their own failures. While these methods outperform supervised fine-tuning, their heavy reliance on predicted trajectories under limited ground-truth supervision leaves them prone to overfitting. To address this, we propose a co-evolving agents framework in which a target agent improves jointly with an auxiliary failure agent. The failure agent learns through preference optimization over failure trajectories from both the target and itself, thereby generating hard negatives that are close to success yet remain failures. Incorporating these informative hard negatives into the target agent's optimization sharpens decision boundaries and enhances generalization. Our comprehensive analysis and experiments across benchmark datasets show that our method not only shows improved performance but also demonstrates that failures, instead of being used as-is, can be systematically transformed into structured and valuable learning signals in self-improving agents.

Figures (3)

• Figure 1: Overview of our co-evolving agents framework. The failure agent learns from pairs of failure trajectories and produces hard negatives, i.e., high-reward failures that lie close to success. These hard negatives are then incorporated into the target agent’s preference optimization, providing more informative contrastive signals. Through this mutual interaction, the two agents co-evolve, enabling the target agent to learn sharper decision boundaries and achieve stronger generalization.
• Figure 2: Diversity analysis
• Figure 3: Ablation on $\lambda_{\text{SFT}}$.