Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards

TL;DR

SynthAgent tackles the data and environment bottlenecks in agentic RL for small LLMs by jointly synthesizing diverse tool-use tasks, stable mock environments, and execution-grounded, rubric-based rewards. A teacher LLM generates underspecified tasks paired with private user context and dedicated tool ecosystems, while a lightweight mock tool/user system ensures stable rollout. Rewards are derived from observable subgoals and interactions rather than subjective judgments, enabling scalable RL across thousands of synthetic tasks. Across 14 challenging datasets, 8B–14B models trained with synthetic data rival or surpass 32B baselines and generalize to short-horizon reasoning, demonstrating cost-efficient practical impact for agentic capabilities.

Abstract

Small LLMs often struggle to match the agentic capabilities of large, costly models. While reinforcement learning can help, progress has been limited by two structural bottlenecks: existing open-source agentic training data are narrow in task variety and easily solved; real-world APIs lack diversity and are unstable for large-scale reinforcement learning rollout processes. We address these challenges with SYNTHAGENT, a framework that jointly synthesizes diverse tool-use training data and simulates complete environments. Specifically, a strong teacher model creates novel tasks and tool ecosystems, then rewrites them into intentionally underspecified instructions. This compels agents to actively query users for missing details. When handling synthetic tasks, an LLM-based user simulator provides user-private information, while a mock tool system delivers stable tool responses. For rewards, task-level rubrics are constructed based on required subgoals, user-agent interactions, and forbidden behaviors. Across 14 challenging datasets in math, search, and tool use, models trained on our synthetic data achieve substantial gains, with small models outperforming larger baselines.

Figures (6)

• Figure 1: Comparison between existing agentic RL training recipes and ours. Open-source agentic training data are narrow in domain, while real-world APIs are costly and unstable. We replace these with diverse synthetic tasks and associated mock environments.
• Figure 2: A unified pipeline for generating synthetic tool-use tasks, constructing stable mock environments, and deriving rubric-based rewards for agentic RL. Diverse tasks and tool ecosystems are created, guided by personas. For each synthetic task, an LLM-simulated user and environment are employed. To assign rewards, multiple trajectories are compared to the previously generated high-level workflow to infer task-specific rubrics.
• Figure 3: Effect of tool-simulator size on TAU-2 performance(5,000 training samples), showing negligible gains from larger simulators.
• Figure 4: Influence of teacher-demonstration count when constructing task-level rubrics(5,000 training samples), indicating that additional demonstrations yield limited gain.
• Figure 5: Impact of increased training data on RL performance, and comparison between RL and SFT at the same data scale.