DockSmith: Scaling Reliable Coding Environments via an Agentic Docker Builder

TL;DR

DockSmith reframes Docker-based environment construction as a core agentic capability rather than a preprocessing step, addressing the brittleness that currently limits scalable execution-grounded training. By deploying a dedicated four-agent pipeline augmented with loop-detection and cross-task memory, and by training on a large corpus of execution-grounded Docker-building trajectories, DockSmith achieves open-source state-of-the-art performance on Multi-Docker-Eval with a 39.72% Fail-to-Pass rate and a 58.28% Commit Rate at 30B scale. The approach also demonstrates transfer to downstream agentic software engineering tasks (SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench 2.0) through joint training with general coding trajectories and carefully designed data filtration and curriculum strategies. Overall, the work provides empirical evidence that environment construction can supply transferable supervision for scaling autonomous software engineering agents and yields a robust dataset of verified environments spanning thousands of repositories for future research.

Abstract

Reliable Docker-based environment construction is a dominant bottleneck for scaling execution-grounded training and evaluation of software engineering agents. We introduce DockSmith, a specialized agentic Docker builder designed to address this challenge. DockSmith treats environment construction not only as a preprocessing step, but as a core agentic capability that exercises long-horizon tool use, dependency reasoning, and failure recovery, yielding supervision that transfers beyond Docker building itself. DockSmith is trained on large-scale, execution-grounded Docker-building trajectories produced by a SWE-Factory-style pipeline augmented with a loop-detection controller and a cross-task success memory. Training a 30B-A3B model on these trajectories achieves open-source state-of-the-art performance on Multi-Docker-Eval, with 39.72% Fail-to-Pass and 58.28% Commit Rate. Moreover, DockSmith improves out-of-distribution performance on SWE-bench Verified, SWE-bench Multilingual, and Terminal-Bench 2.0, demonstrating broader agentic benefits of environment construction.

Figures (4)

• Figure 1: DockSmith training framework. A multi-agent pipeline with four agents (Context Retrieval, Dockerfile, Eval Script, Test Analysis) generates execution-grounded Docker-building trajectories from test-backed pull requests in an iterative repair loop. Loop detection and cross-task success memory are used, followed by trajectory filtering and curriculum sampling for supervised fine-tuning.
• Figure 2: Impact of mixing Docker-building trajectories during training. Mixing ratios are defined at the token level and written as $\text{SWE}:\text{Docker}$: for example, 1:0.25 means adding 0.25 Docker-building tokens per SWE token, while keeping the SWE token budget fixed. Bars report $\Delta$ scores relative to an SWE-only baseline on SWE.V, SWE.M, and Terminal-Bench 2.0.
• Figure 3: Docker-building error counts by error type on Multi-Docker-Eval (aggregated over all trajectories), comparing the baseline and DockSmith. Percent annotations indicate relative changes.
• Figure 4: Error propagation probability matrices aggregated across SWE-bench Verified, SWE-bench Multilingual, and Terminal-Bench 2.0. Each cell shows the probability that an error at the row layer transitions to an error at the column layer in the next step. Left: Baseline. Center: DockSmith. Right: Difference ($\Delta$), where blue indicates lower propagation and red indicates higher propagation.