AutoBaxBuilder: Bootstrapping Code Security Benchmarking

TL;DR

AutoBaxBuilder automates the generation of novel, end-to-end code-security benchmarks by leveraging an LLM-driven pipeline to create scenarios, functional tests, and exploits with minimal human input. Building on BaxBench, it validates test and exploit quality and expands the benchmark with AutoBaxBench across Easy/Medium/Hard subsets, enabling scalable, low-cost evaluation of state-of-the-art models. The approach reproduces known trends while often yielding more thorough security testing, highlighting both the potential and challenges of automated benchmark generation, including CWE-400 caveats and model-bias considerations. Collectively, the work demonstrates the feasibility of sustained, scalable security evaluation for LLM-based code and outlines practical paths toward extending benchmarks to broader domains and evaluation settings.

Abstract

As LLMs see wide adoption in software engineering, the reliable assessment of the correctness and security of LLM-generated code is crucial. Notably, prior work has demonstrated that security is often overlooked, exposing that LLMs are prone to generating code with security vulnerabilities. These insights were enabled by specialized benchmarks, crafted through significant manual effort by security experts. However, relying on manually-crafted benchmarks is insufficient in the long term, because benchmarks (i) naturally end up contaminating training data, (ii) must extend to new tasks to provide a more complete picture, and (iii) must increase in difficulty to challenge more capable LLMs. In this work, we address these challenges and present AutoBaxBuilder, a framework that generates tasks and tests for code security benchmarking from scratch. We introduce a robust pipeline with fine-grained plausibility checks, leveraging the code understanding capabilities of LLMs to construct functionality tests and end-to-end security-probing exploits. To confirm the quality of the generated benchmark, we conduct both a qualitative analysis and perform quantitative experiments, comparing it against tasks constructed by human experts. We use AutoBaxBuilder to construct entirely new tasks and release them to the public as AutoBaxBench, together with a thorough evaluation of the security capabilities of LLMs on these tasks. We find that a new task can be generated in under 2 hours, costing less than USD 10.

Figures (29)

• Figure 1: Overview of our method. The LLM-based pipeline starts from scratch and produces a complete benchmark instance with scenario description ①, test cases ②, and end-to-end exploits ③. After generating a novel scenario description, the LLM generates functional tests and solutions, iterating until execution feedback confirms that the tests are correct. Next, the LLM designs end-to-end exploits to expose vulnerabilities, iterating until it finds a pair of solutions, one on which the exploit succeeds and one on which it fails. The results are combined into a new task instance.
• Figure 1: Overview over key statistics of AutoBaxBench, showing the overall benchmark and its Easy to Hard subsets in comparison to BaxBench.
• Figure 2: Flag system for RefineExploit
• Figure 3: LLM performance comparison on scenarios from BaxBench, with human-written tests in red, and tests written by our method AutoBaxBuilder in blue. Functional correctness trends are highly similar, while security tests by AutoBaxBuilder are stricter and have higher coverage.
• Figure 4: Confusion matrix on pass@1 between BaxBench and AutoBaxBench, showing high correlation.