Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents

TL;DR

Trajectory2Task introduces a general, verifiable pipeline for synthesizing multi-turn tool-use data under ambiguous, changing, and infeasible user intents. By generating executable tool trajectories first and deriving corresponding user-facing tasks, it enables closed-loop evaluation and targeted fine-tuning. In Retail-3I, seven state-of-the-art LLMs reveal robustness gaps under complex intents, while trajectory-based supervised fine-tuning on 2,872 trajectories yields consistent performance gains and cross-domain transfer. The work demonstrates that enabling online plan revision and constrained decision-making through synthesized data leads to more robust tool-calling behavior and better generalization to new domains. Limitations include reliance on simulated users and the Tau2-Bench environment, suggesting future work to expand domain coverage and human-grounded validation.

Abstract

Tool-calling agents are increasingly deployed in real-world customer-facing workflows. Yet most studies on tool-calling agents focus on idealized settings with general, fixed, and well-specified tasks. In real-world applications, user requests are often (1) ambiguous, (2) changing over time, or (3) infeasible due to policy constraints, and training and evaluation data that cover these diverse, complex interaction patterns remain under-represented. To bridge the gap, we present Trajectory2Task, a verifiable data generation pipeline for studying tool use at scale under three realistic user scenarios: ambiguous intent, changing intent, and infeasible intents. The pipeline first conducts multi-turn exploration to produce valid tool-call trajectories. It then converts these trajectories into user-facing tasks with controlled intent adaptations. This process yields verifiable task that support closed-loop evaluation and training. We benchmark seven state-of-the-art LLMs on the generated complex user scenario tasks and observe frequent failures. Finally, using successful trajectories obtained from task rollouts, we fine-tune lightweight LLMs and find consistent improvements across all three conditions, along with better generalization to unseen tool-use domains, indicating stronger general tool-calling ability.

Figures (2)

• Figure 1: A real-world tool-calling dialogue with complex user intent. In real-world tool-calling scenarios, the user intents are often ambiguous, changing, or even infeasible, requiring the agent to reason over partial information, ask clarifying questions, adapt plans, and handle unsupported requests.
• Figure 2: Trajectory2Task: a two-stage verifiable data generation pipeline. (1) Trajectory Exploration: A powerful tool-calling LLM agent (Claude-4.5-Sonnet) leverages sampled user information, trajectory examples, and tool subset from the API graph as context, then performs self-exploration in the environment to produce exploratory trajectories. (2) Task Generation: Filtered trajectories are transformed into realistic tasks with user intent adaptation, including ambiguous intent, changing intent, and infeasible intent. The generated task are further validated by an LLM to ensure quality. The valid tasks are naturally paired with golden label trajectories (i.e., the exploratory trajectory). We use the generate data to further train and evaluate LLMs.