From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents

TL;DR

The work tackles the data bottleneck and noisy RL signals in post-training, interactive tool-using agents by introducing EigenData, a self-evolving data engine that jointly designs workflows, prompts, and executable verifiers. It pairs EigenData with a verifier-based RL recipe (GRPO-based) that uses trajectory-level group-relative advantages and dynamic filtering to stabilize learning, aided by a finely-tuned user simulator. Empirically, the approach delivers strong results on tau^2-bench across Airline, Retail, and Telecom, with the 235B model reaching state-of-the-art or frontier-level performance while maintaining open-weight accessibility. The framework demonstrates a scalable, execution-grounded pathway to developing complex tool-using agents with reduced human annotation requirements, while highlighting considerations for safety and policy adherence in deployment.

Abstract

Interactive tool-using agents must solve real-world tasks via multi-turn interaction with both humans and external environments, requiring dialogue state tracking, multi-step tool execution, while following complex instructions. Post-training such agents is challenging because synthesis for high-quality multi-turn tool-use data is difficult to scale, and reinforcement learning (RL) could face noisy signals caused by user simulation, leading to degraded training efficiency. We propose a unified framework that combines a self-evolving data agent with verifier-based RL. Our system, EigenData, is a hierarchical multi-agent engine that synthesizes tool-grounded dialogues together with executable per-instance checkers, and improves generation reliability via closed-loop self-evolving process that updates prompts and workflow. Building on the synthetic data, we develop an RL recipe that first fine-tunes the user model and then applies GRPO-style training with trajectory-level group-relative advantages and dynamic filtering, yielding consistent improvements beyond SFT. Evaluated on tau^2-bench, our best model reaches 73.0% pass^1 on Airline and 98.3% pass^1 on Telecom, matching or exceeding frontier models. Overall, our results suggest a scalable pathway for bootstrapping complex tool-using behaviors without expensive human annotation.

Figures (5)

• Figure 1: EigenData, a hierarchical self-evolving multi-agent framework for agentic data generation. It comprises two layers: a top-level orchestration layer responsible for planning workflow, writing prompts, and quality control, and an execution layer (the figure shows an example workflow for agentic data generation planned by the workflow planner agent) composed of worker agents that perform domain-specific generation and validation with the workflow planned by orchestration layer.
• Figure 1: RL training curves for separate training.
• Figure 2: Impact of user model quality. Both scenarios feature identical user instructions. In (a), the base user model ignores the agent's instruction and exhausts irrelevant tools, causing task failure. In (b), the SFT-trained user model correctly interprets and executes the tool, enabling task success. User simulation errors can corrupt RL reward signals, penalizing correct agent behavior.
• Figure 2: RL training curves for mix training.
• Figure 3: User Model Ablation on Telecom Domain. We compare RL training with different user models: without SFT (Qwen3-30B-A3B-2507 as user simulator) vs. with SFT (fine-tuned Qwen3-30B-A3B-2507). Training with a low-quality user model degrades performance, while training with the fine-tuned user model improves performance.