Self-Challenging Language Model Agents

TL;DR

The paper introduces Self-Challenging Agent (SCA), a framework where an LLM alternates between challenging itself to generate high-quality, verifiable tasks (Code-as-Task, CaT) and solving those tasks to improve its tool-use capabilities. CaT formalizes tasks with an instruction, a verification function, an example solution, and failure cases to ensure feasibility and difficulty, aided by external code execution for automatic filtering. Through distillation and self-improvement experiments on M3ToolEval and TauBench, SCA achieves substantial gains over baselines like PAE and zero-shot models, demonstrating the value of autonomous task synthesis for open-ended environments. Ablation and scaling analyses show CaT reduces false positives/negatives, while larger task sets and diverse trajectories further boost performance, albeit with remaining limitations in achieving broad, environment-general agentic capabilities. Overall, SCA offers a scalable flywheel for self-improvement of multi-turn tool-use LLM agents, with CaT as a pivotal mechanism for task quality control.

Abstract

Large language models are quickly becoming the foundation for intelligent agents that are capable of using tools. However, training such agents is challenging because it requires human creation and annotation of a diverse set of tasks, tools, and evaluation criteria. In this paper, we propose the Self-Challenging framework for training an agent on high-quality tasks that are generated by itself. The agent first plays the role of challenger and generates a task after interacting with the given tools. The tasks take the form of a novel general class of problems termed Code-as-Task, which are defined by an instruction, a verification function and solution and failure cases which serve as tests, allowing to filter only for high-quality tasks. The agent then takes an executor role and trains on those tasks with reinforcement learning using the evaluation feedback as a reward. Evaluation on two existing multi-turn tool-use agent benchmarks, M3ToolEval and TauBench, shows the Self-Challenging framework achieves over a two-fold improvement in Llama-3.1-8B-Instruct, despite using only self-generated training data.

Figures (20)

• Figure 1: Overview of Self-Challenging Agent. The agent takes on two roles: task challenger, and task executor. The task challenger proposes a task along with a verification method to verify the solution to the task. The task executor generates a solution and obtains a reward from the environment based on the verification method.
• Figure 2: An example of a synthetic Code-as-Task (CaT) generated by the task challenger, in a TauBench-based environment yao2024taubenchbenchmarktoolagentuserinteraction. The task challenger interacts with the environment taking a series of actions calling different tools to gather information, before generating the synthetic task, consisting of an instruction, verification function, example solution and failure cases (shortened in the figure for brevity). Automatic filtering is applied to CaTs to keep only valid tasks where the example solution can pass the verification function and the failure cases cannot.
• Figure 3: Ablation studies of different RL algorithms with synthetic tasks generated from SCA, in the Calculation environment from M$^3$ToolEval. Pass@1 success rates are reported. We find that online RL algorithms in general attain even better performance on out-of-distribution test sets, but they are more unstable and require more careful tuning.
• Figure 4: Human annotations of synthetic task qualities. 50 rollout trajectories from Llama-3.1-8B in the Retail environment from attempted synthetic tasks from each variant are manually labeled to fall into one of the four categories including False Negative (FN), False Positive (FP), True Negative (TN), and True Positive (TP). The pass rates of the task challenger generating a task passing all filters for each category are shown in parentheses. We observe that CaT can significantly reduce both FN and FP, which are invalid tasks or wrongly labeled trajectories.
• Figure 5: Analysis of the distribution of task difficulty before and after the filtering step of CaT in the Retail environment. The task difficulty is represented by the length of the example solution. The percentages of passing tasks after filtering are included in parentheses. We observe that CaT filtering can result in a less diverse task distribution for the less capable Llama-3.1-8B but preserves the original task distribution for the stronger Llama-3.1-70B model.