ASTRA: Automated Synthesis of agentic Trajectories and Reinforcement Arenas

TL;DR

ASTRA addresses the challenge of training robust tool-augmented LLM agents by introducing a fully automated, end-to-end framework that combines scalable data synthesis with verifiable multi-turn RL. It pairs a trajectory-synthesis pipeline grounded in static tool-call graphs for supervised fine-tuning with an environment-synthesis framework that produces code-executable, rule-verifiable environments for online RL, enabling long-horizon decision making. A two-stage training regime—SFT followed by multi-turn RL—yields state-of-the-art results on agentic benchmarks at comparable scales, while preserving core reasoning ability; the approach hinges on an F1-style trajectory reward that balances task completion and interaction efficiency. The authors release the complete pipelines, environments, and trained models to support reproducibility and future research, signaling practical impact for scalable, reliable tool-using agents across domains.

Abstract

Large language models (LLMs) are increasingly used as tool-augmented agents for multi-step decision making, yet training robust tool-using agents remains challenging. Existing methods still require manual intervention, depend on non-verifiable simulated environments, rely exclusively on either supervised fine-tuning (SFT) or reinforcement learning (RL), and struggle with stable long-horizon, multi-turn learning. To address these challenges, we introduce ASTRA, a fully automated end-to-end framework for training tool-augmented language model agents via scalable data synthesis and verifiable reinforcement learning. ASTRA integrates two complementary components. First, a pipeline that leverages the static topology of tool-call graphs synthesizes diverse, structurally grounded trajectories, instilling broad and transferable tool-use competence. Second, an environment synthesis framework that captures the rich, compositional topology of human semantic reasoning converts decomposed question-answer traces into independent, code-executable, and rule-verifiable environments, enabling deterministic multi-turn RL. Based on this method, we develop a unified training methodology that integrates SFT with online RL using trajectory-level rewards to balance task completion and interaction efficiency. Experiments on multiple agentic tool-use benchmarks demonstrate that ASTRA-trained models achieve state-of-the-art performance at comparable scales, approaching closed-source systems while preserving core reasoning ability. We release the full pipelines, environments, and trained models at https://github.com/LianjiaTech/astra.

Figures (13)

• Figure 1: Comparison of Model Performance on BFCL v3 Multi-Turn.
• Figure 2: Overview of the Tool-Chain-Based Trajectory Synthesis Pipeline.
• Figure 3: Overview of the QA-Based Environment Synthesis Framework.
• Figure 4: Rollout Procedure in Reinforcement Learning.
• Figure 5: One-Step Adaptive Batch Filling.