Endless Terminals: Scaling RL Environments for Terminal Agents

TL;DR

This work tackles the environment bottleneck in training terminal agents by introducing Endless Terminals, a fully autonomous, four-phase procedural generation pipeline that creates and verifies thousands of terminal-use tasks with zero human labeling. The approach couples diverse, automatically verifiable task generation with a minimalist PPO training loop, yielding substantial improvements across multiple models and transferring to human-curated benchmarks like TerminalBench 2.0. Key contributions include a scalable, end-to-end task generator, automatic solvability filtering via $pass@16$, and evidence that simple RL can scale when the environment does, achieving meaningful gains without complex agentic scaffolds. The results imply that investing in automated, verifiable environment generation can unlock robust RL training for real-world terminal tasks, with implications for scalable, reusable benchmarks and deployment-ready agents.

Abstract

Environments are the bottleneck for self-improving agents. Current terminal benchmarks were built for evaluation, not training; reinforcement learning requires a scalable pipeline, not just a dataset. We introduce Endless Terminals, a fully autonomous pipeline that procedurally generates terminal-use tasks without human annotation. The pipeline has four stages: generating diverse task descriptions, building and validating containerized environments, producing completion tests, and filtering for solvability. From this pipeline we obtain 3255 tasks spanning file operations, log management, data processing, scripting, and database operations. We train agents using vanilla PPO with binary episode level rewards and a minimal interaction loop: no retrieval, multi-agent coordination, or specialized tools. Despite this simplicity, models trained on Endless Terminals show substantial gains: on our held-out dev set, Llama-3.2-3B improves from 4.0% to 18.2%, Qwen2.5-7B from 10.7% to 53.3%, and Qwen3-8B-openthinker-sft from 42.6% to 59.0%. These improvements transfer to human-curated benchmarks: models trained on Endless Terminals show substantial gains on held out human curated benchmarks: on TerminalBench 2.0, Llama-3.2-3B improves from 0.0% to 2.2%, Qwen2.5-7B from 2.2% to 3.4%, and Qwen3-8B-openthinker-sft from 1.1% to 6.7%, in each case outperforming alternative approaches including models with more complex agentic scaffolds. These results demonstrate that simple RL succeeds when environments scale.

Figures (7)

• Figure 1: Endless Terminals. Tasks are procedurally generated through four phases: (I) task description generation, (II) container setup with iterative validation, (III) completion test generation, and (IV) solution-based filtering using o3. The pipeline yields 3255 verified tasks for training terminal agents with PPO.
• Figure 2: Overview of the Endless Terminals procedural generation pipeline. Phase I generates diverse task descriptions by sampling across categories, complexity levels, and scenario contexts, producing both a task description and privileged ground truth data for verification. Phase II creates containerized environments and validates them with self written prerequisite tests. Phase III generates completion tests that verify the expected end state. Phase IV filters tasks by sampling 16 solutions from o3, retaining only tasks where at least one solution succeeds. Some example tasks include shell configuration, log analysis, and checksum generation.
• Figure 3: Training and evaluation results. Top row: Reward curves during PPO training on Endless Terminals for (left) Llama-3.2-3B, (center) Qwen2.5-7B, and (right) Qwen3-8B-openthinker-sft, showing consistent improvement across all models. Bottom row: Pass rates on (left) our development set, (center) OpenThinker development set , and (right) TerminalBench 2.0 . Models trained with our RL approach (+RL Ours) outperform base models and alternative finetuned variants across all evaluations. Here, RL (OpenThinker) denotes RL training in OpenThoughts-Agent openthoughts-agent. Results on Terminal Bench 2.0 are averaged over 5 runs for our methods.
• Figure 4: Distribution of tasks in Endless Terminals. Left: Task categories, with file operations and log management comprising the largest shares.
• Figure 5: Failure Analysis. Failure modes are dominated by loop behaviors (39%) and turn exhaustion (26%), with remaining failures terminating early on specialized domains like cryptanalysis and bioinformatics (right).