AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework

TL;DR

AgentRL tackles the challenge of scaling agentic reinforcement learning for large language models in multi-turn, multi-task settings. It combines an asynchronous rollout-training pipeline, a unified function-call based environment API, containerized task environments, and a centralized controller with two key algorithms: cross-policy sampling and task advantage normalization. Empirical results show state-of-the-art performance across five agentic tasks and strong generalization to unseen tasks, with multi-task training matching the best of single-task specialists. The work demonstrates a practical path toward generalist, tool-using LLM agents and informs future improvements in scalable, heterogeneous RL for language models.

Abstract

Recent advances in large language models (LLMs) have sparked growing interest in building generalist agents that can learn through online interactions. However, applying reinforcement learning (RL) to train LLM agents in multi-turn, multi-task settings remains challenging due to lack of scalable infrastructure and stable training algorithms. In this work, we present the AgentRL framework for scalable multi-turn, multi-task agentic RL training. On the infrastructure side, AgentRL features a fully-asynchronous generation-training pipeline for efficient multi-turn RL. To support heterogeneous environment development in multi-task RL, we design a unified function-call based API interface, containerized environment development, and a centralized controller. On the algorithm side, we propose cross-policy sampling to encourage model exploration in multi-turn settings and task advantage normalization to stabilize multi-task training. Experiments show that AgentRL, trained on open LLMs across five agentic tasks, significantly outperforms GPT-5, Clause-Sonnet-4, DeepSeek-R1, and other open-source LLM agents. Multi-task training with AgentRL matches the best results among all task-specific models. AgentRL is open-sourced at https://github.com/THUDM/AgentRL. The algorithm and framework are adopted in building \textsc{\href{https://autoglm.zhipuai.cn}{AutoGLM}}.

Figures (8)

• Figure 1: Overall performance of AgentRL.
• Figure 2: An overview of AgentRL. Top: asynchronous training and rollout flows. Bottom: the environment framework where a controller manages multiple workers to provide environments, and the rollout details, including cross-policy sampling and task advantage normalization.
• Figure 3: Synchronous vs. Asynchronous Training. The asynchronous design improves efficiency by separating data rollout and model training on different resource groups.
• Figure 4: Throughput of AgentRL vs. the synchronous baseline for 14B parameter (Qwen2.5) models on Webshop (log-scale for both axes).
• Figure 5: The AgentRL training pipeline, decoupled into a Training Framework and an Environment Deployment Framework, organized by a central AgentRL Controller. The Training Framework is responsible for policy rollouts and updates, while the Environment Deployment Framework manages scalable, containerized task environments that provide feedback.