Exploring Expert Failures Improves LLM Agent Tuning

TL;DR

This paper addresses the limitation of Rejection Sampling Fine-Tuning (RFT) where expert trajectories often fail on hard, out-of-distribution subtasks. It introduces Exploring Expert Failures (EEF), a three-phase framework that learns from beneficial actions hidden in failed trajectories via simulation, selective training, and recovery-focused state handling. Empirical results on WebShop and SciWorld show that EEF substantially outperforms baselines, including GPT-4 and RFT variants, achieving state-of-the-art scores and demonstrating robustness across models and data regimes. The work suggests that leveraging negative demonstrations, tempered by careful action selection and recovery strategies, can markedly improve LLM agent tuning with no reward-model overhead, offering practical implications for scalable, cost-efficient agent training.

Abstract

Large Language Models (LLMs) have shown tremendous potential as agents, excelling at tasks that require multiple rounds of reasoning and interactions. Rejection Sampling Fine-Tuning (RFT) has emerged as an effective method for finetuning LLMs as agents: it first imitates expert-generated successful trajectories and further improves agentic skills through iterative fine-tuning on successful, self-generated trajectories. However, since the expert (e.g., GPT-4) succeeds primarily on simpler subtasks and RFT inherently favors simpler scenarios, many complex subtasks remain unsolved and persistently out-of-distribution (OOD). Upon investigating these challenging subtasks, we discovered that previously failed expert trajectories can often provide valuable guidance, e.g., plans and key actions, that can significantly improve agent exploration efficiency and acquisition of critical skills. Motivated by these observations, we propose Exploring Expert Failures (EEF), which identifies beneficial actions from failed expert trajectories and integrates them into the training dataset. Potentially harmful actions are meticulously excluded to prevent contamination of the model learning process. By leveraging the beneficial actions in expert failures, EEF successfully solves some previously unsolvable subtasks and improves agent tuning performance. Remarkably, our approach achieved a 62\% win rate in WebShop, outperforming RFT (53. 6\%) and GPT-4 (35. 6\%), and to the best of our knowledge, setting a new state-of-the-art as the first method to surpass a score of 0.81 in WebShop and exceed 81 in SciWorld.

Figures (3)

• Figure 1: $\tau_e=[s_0, s_1, \dots]$ is a failed expert (GPT-4) trajectory for a challenging subtask $s_0$ with $R(\tau_e) = 0$. Despite its overall failure, $\tau_e$ contains partial insights to train agent $\pi_\theta$. To identify beneficial actions in $\tau_e$, our method (EEF) performs simulations from expert states $[s_0, s_l, \dots]$ at intervals of length $l$, resulting in $[\tau_{s_0}, \tau_{s_l}, \dots]$, revealing that certain expert actions $a_{0:l-1}$ and $a_{2l:3l-1}$ enable the agent to transit from failed states $s_0, s_{2l}$ to positively-performing states $s_l, s_{3l}$. EEF thus considers these actions (green arrows) beneficial for solving $s_0, s_{2l}$ and retains them in the SFT training dataset while rejecting the remaining actions (red arrows).
• Figure 2: The percentage of successful subtasks that involve the use of navigation skills: Next and Back.
• Figure 3: Win rates of different methods training the same base model for one iteration under varying simulation budgets (exploration cost) by adjusting $N$ or $M$. RFT × N: Initial state $s_0$ simulated $N$ times. EEF: Different expert state simulation numbers $M=1,2,5$ on different expert datasets.