BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software

TL;DR

The paper tackles the challenge of automatically compiling diverse, real-world OSS by introducing Build-Bench, a statistically representative benchmark with manually labeled ground-truth build instructions and binaries. It proposes OSS-Build-Agent, a strong, model-agnostic agent that combines LLM-assisted retrieval with a two-agent compilation loop to extend build knowledge and iteratively fix compilation errors. Empirical results show that agent-based approaches substantially outperform rule-based baselines, with performance improving further when leveraging stronger LLMs and retrieval strategies, though instability and error resolution remain important challenges. The work offers detailed design analyses and emphasizes the benchmark's value for driving advances in automated software engineering and security workflows.

Abstract

Automatically compiling open-source software (OSS) projects is a vital, labor-intensive, and complex task, which makes it a good challenge for LLM Agents. Existing methods rely on manually curated rules and workflows, which cannot adapt to OSS that requires customized configuration or environment setup. Recent attempts using Large Language Models (LLMs) used selective evaluation on a subset of highly rated OSS, a practice that underestimates the realistic challenges of OSS compilation. In practice, compilation instructions are often absent, dependencies are undocumented, and successful builds may even require patching source files or modifying build scripts. We propose a more challenging and realistic benchmark, BUILD-BENCH, comprising OSS that are more diverse in quality, scale, and characteristics. Furthermore, we propose a strong baseline LLM-based agent, OSS-BUILD-AGENT, an effective system with enhanced build instruction retrieval module that achieves state-of-the-art performance on BUILD-BENCH and is adaptable to heterogeneous OSS characteristics. We also provide detailed analysis regarding different compilation method design choices and their influence to the whole task, offering insights to guide future advances. We believe performance on BUILD-BENCH can faithfully reflect an agent's ability to tackle compilation as a complex software engineering tasks, and, as such, our benchmark will spur innovation with a significant impact on downstream applications in the fields of software development and software security.

Figures (5)

• Figure 1: Demonstration of rule-based and AI agentic compilation methods. While rule-based methods follow a predefined workflow, they cannot adequately adapt to different environments. In comparison, AI agents leverage their pre-trained knowledge to adjust the compilation commands based on execution results. In this example, the agent realizes libevent is a key missing dependency for Bitcoin to compile and installs it to successfully compile the project.
• Figure 2: Distribution of stargazer counts of Build-Bench and CompileAgentBench. Note that in Build-Bench, relatively low-profile repositories made up the majority of the samples.
• Figure 3: OSS-Build-Agent system diagram. The initial input is the README, then an optional LLM extends this with additional compilation instructions. Finally, a multi-agent build system iteratively generates and executes compilation steps, attempting to compile the target repository.
• Figure 4: pass@k performance of OSS-Build-Agent with LLM-Assisted Retrieval using GPT-4o. For $K=1$, we report the earliest chronological run.
• Figure 5: Agent failure modes analysis of OSS-Build-Agent enhanced with LLM-Assisted Retrieval implemented with o3-mini.