Generalizable End-to-End Tool-Use RL with Synthetic CodeGym

Abstract

Tool-augmented large language models (LLMs), hereafter LLM agents, leverage external tools to solve diverse tasks and interface with the real world. However, current training practices largely rely on supervised fine-tuning (SFT) over static trajectories or reinforcement learning (RL) on narrow tasks, which generalize poorly beyond development settings and lead to brittleness with new tools and unseen workflows. Because code execution reflects many structural patterns of real-world workflows, we use coding problems as a structured substrate to build tool-use agent training environments with diverse task configurations. To this end, we introduce CodeGym, a scalable framework that synthesizes diverse, verifiable, and controllable multi-turn tool-use environments for agent RL, enabling LLM agents to explore and master various workflows actively. CodeGym converts static coding problems into interactive environments by extracting atomic functions or logic into callable tools, yielding verifiable tasks that span various tool-execution workflows. Models of varying sizes and chain-of-thought configurations trained in CodeGym exhibit consistent out-of-distribution generalizability; for example, Qwen2.5-32B-Instruct achieves an absolute accuracy gain of 8.7 points on the OOD benchmark $τ$-Bench. These results highlight CodeGym as a step toward scalable general-purpose RL environments for training tool-use behaviors that align with real-world agent workflows.

Figures (18)

• Figure 1: Overview of CodeGym. We transform coding problems into interactive environments to train LLM agents. (Left) We extract atomic and reusable functions or logic from coding solutions to construct interactive environments. (Middle) CodeGym enables agents to solve tasks via multi-turn tool calls, with environment correctness verified automatically. (Right) The resulting environments support scalable RL training, improving robustness and generalization of LLM agents.
• Figure 2: Pipeline for CodeGym Environment Generation. Coding problems are reformulated into interactive environments by extracting tools, generating candidate solutions, and validating them with unit tests. The environment is deemed valid if any candidate solution passes all tests, and the resulting unit tests serve as task configurations for RL training.
• Figure 3: CodeGym Environment Example. Given the problem description and the action list, the agent interactively solves the task and receives a binary reward after submitting the answer.
• Figure 4: CodeGym Statistics. The average numbers of tools and steps to solve tasks are 6.52 and 44.07, respectively, indicating that CodeGym encompasses diverse tools and complex logic.
• Figure 5: RL Training Pipeline for CodeGym. A server provides centralized control of environments, and each rollout process is allocated to a service port. The rollout workers send actions to the corresponding service ports and receive observations. The rollout controller sends commands to initialize the environments and receive reward signals to form the replay buffer.