CodeMonkeys: Scaling Test-Time Compute for Software Engineering

TL;DR

CodeMonkeys investigates scaling test-time compute to improve real-world software engineering problem solving on SWE-bench. It introduces a three-state-machine framework (Testing, Editing, Selection) that jointly generates edits and executable tests, enabling both serial and parallel scaling across issues. By amortizing codebase context via a codebase-wide relevance scan and applying a three-stage pipeline (context identification, candidate generation, and selection), it achieves 69.8% problem coverage and a 57.4% final SWE-bench Verified score with roughly $2300 in inference costs; an ensemble variant, Barrel of Monkeys, reaches 80.8% coverage and 66.2% final score, illustrating the potential of combining diverse sources. The work highlights limitations in context recall, generation diversity, and selection strategies, and outlines future directions such as richer feedback, longer context integration, and enhanced selection to further close the gap to oracle performance.

Abstract

Scaling test-time compute is a promising axis for improving LLM capabilities. However, test-time compute can be scaled in a variety of ways, and effectively combining different approaches remains an active area of research. Here, we explore this problem in the context of solving real-world GitHub issues from the SWE-bench dataset. Our system, named CodeMonkeys, allows models to iteratively edit a codebase by jointly generating and running a testing script alongside their draft edit. We sample many of these multi-turn trajectories for every issue to generate a collection of candidate edits. This approach lets us scale "serial" test-time compute by increasing the number of iterations per trajectory and "parallel" test-time compute by increasing the number of trajectories per problem. With parallel scaling, we can amortize up-front costs across multiple downstream samples, allowing us to identify relevant codebase context using the simple method of letting an LLM read every file. In order to select between candidate edits, we combine voting using model-generated tests with a final multi-turn trajectory dedicated to selection. Overall, CodeMonkeys resolves 57.4% of issues from SWE-bench Verified using a budget of approximately 2300 USD. Our selection method can also be used to combine candidates from different sources. Selecting over an ensemble of edits from existing top SWE-bench Verified submissions obtains a score of 66.2% and outperforms the best member of the ensemble on its own. We fully release our code and data at https://scalingintelligence.stanford.edu/pubs/codemonkeys.

Figures (7)

• Figure 1: Overview of the CodeMonkeys system. Left: We retrieve codebase context by using models to first identify relevant files and then rank them relative to each other. Middle: We generate a codebase edit and testing script using a pair of multi-turn state machines that iterate based on execution feedback. We run these state machines multiple times in parallel to generate 10 edits and tests for every issue. Right: We select between candidate edits by identifying the candidates that pass the most generated tests and asking a model to decide between these top candidates. For details about our system's three state machines, see Figure \ref{['fig:sm_detail']}.
• Figure 2: Measuring CodeMonkeys performance across the three subtasks we identify in Section \ref{['sec:method']} (context, generation, and selection). Note that modifying the approach to one subtask can influence the performance on other subtasks as well. For example, generating more candidate edits could increase coverage but make selection harder.
• Figure 3: Left: Measuring recall (the fraction of SWE-bench problems with context windows that contain all needed files) as we increase the context window size limit. With the 128k token limit that we use for later experiments, 92.6% of instances have the correct files in context. Right: Visualizing the distribution of context compression factors across SWE-bench problems, i.e. the ratio between the cumulative token count of files scanned by the relevance model and the cumulative token count of files we include after relevance + ranking.
• Figure 4: Details of the CodeMonkeys state machines. The Testing State Machine iteratively generates an initial draft of a testing script based on execution feedback from running the test on the codebase before any edits are applied. The Editing State Machine first generates an initial edit conditioned on the codebase context and the output test of the Testing State Machine. Then, it refines both the test and edit draft based on execution feedback from running the test before and after the edit is applied. The Selection State Machine first generates a test to distinguish between the top 3 candidate edits that pass the most testing scripts. Then, based on execution feedback of running this test with all of the candidate edits and on the codebase without edits, chooses to either create a new test script to further differentiate between the edits or selects a final edit.
• Figure 5: Measuring coverage (left) and score when using majority-voting selection (right) as we sweep over the number of serial iterations per editing state machine and the number of parallel state machines sampled per problem. Each colored curve corresponds to a different number of parallel state machines, and the dots along each curve correspond to increased numbers of sequential iterations per state machine. The first few serial iterations have a large impact on improving performance. However, past that point, different configurations with similar costs lead to similar performance, particularly for coverage.