Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OSS-Bench: Benchmark Generator for Coding LLMs

Yuancheng Jiang, Roland Yap, Zhenkai Liang

TL;DR

OSS-Bench introduces an automatic, live benchmark generator that derives evaluation tasks from real open-source software by replacing functions with LLM-generated variants and assessing compilability, functional correctness, and memory safety via robust ground-truth signals. The framework emphasizes live evolution, multi-language applicability, and automated ground truth without human or LLM-generated annotations. Instantiations on the PHP interpreter and SQLite3 engine demonstrate the system's ability to profile 17 LLMs, reveal non-obvious patterns such as intra-family differences and size-performance gaps, and mitigate overfitting through dynamic task evolution. Extended fuzzing further exposes memory-safety weaknesses in LLM edits, underscoring the value of a scalable, automated, memory-safety–aware benchmark for real-world coding tasks.

Abstract

In light of the rapid adoption of AI coding assistants, LLM-assisted development has become increasingly prevalent, creating an urgent need for robust evaluation of generated code quality. Existing benchmarks often require extensive manual effort to create static datasets, rely on indirect or insufficiently challenging tasks, depend on non-scalable ground truth, or neglect critical low-level security evaluations, particularly memory-safety issues. In this work, we introduce OSS-Bench, a benchmark generator that automatically constructs large-scale, live evaluation tasks from real-world open-source software. OSS-Bench replaces functions with LLM-generated code and evaluates them using three natural metrics: compilability, functional correctness, and memory safety, leveraging robust signals like compilation failures, test-suite violations, and sanitizer alerts as ground truth. In our evaluation, the benchmark, instantiated as OSS-Bench(php) and OSS-Bench(sql), profiles 17 diverse LLMs, revealing insights such as intra-family behavioral patterns and inconsistencies between model size and performance. Our results demonstrate that OSS-Bench mitigates overfitting by leveraging the evolving complexity of OSS and highlights LLMs' limited understanding of low-level code security via extended fuzzing experiments. Overall, OSS-Bench offers a practical and scalable framework for benchmarking the real-world coding capabilities of LLMs.

OSS-Bench: Benchmark Generator for Coding LLMs

TL;DR

OSS-Bench introduces an automatic, live benchmark generator that derives evaluation tasks from real open-source software by replacing functions with LLM-generated variants and assessing compilability, functional correctness, and memory safety via robust ground-truth signals. The framework emphasizes live evolution, multi-language applicability, and automated ground truth without human or LLM-generated annotations. Instantiations on the PHP interpreter and SQLite3 engine demonstrate the system's ability to profile 17 LLMs, reveal non-obvious patterns such as intra-family differences and size-performance gaps, and mitigate overfitting through dynamic task evolution. Extended fuzzing further exposes memory-safety weaknesses in LLM edits, underscoring the value of a scalable, automated, memory-safety–aware benchmark for real-world coding tasks.

Abstract

In light of the rapid adoption of AI coding assistants, LLM-assisted development has become increasingly prevalent, creating an urgent need for robust evaluation of generated code quality. Existing benchmarks often require extensive manual effort to create static datasets, rely on indirect or insufficiently challenging tasks, depend on non-scalable ground truth, or neglect critical low-level security evaluations, particularly memory-safety issues. In this work, we introduce OSS-Bench, a benchmark generator that automatically constructs large-scale, live evaluation tasks from real-world open-source software. OSS-Bench replaces functions with LLM-generated code and evaluates them using three natural metrics: compilability, functional correctness, and memory safety, leveraging robust signals like compilation failures, test-suite violations, and sanitizer alerts as ground truth. In our evaluation, the benchmark, instantiated as OSS-Bench(php) and OSS-Bench(sql), profiles 17 diverse LLMs, revealing insights such as intra-family behavioral patterns and inconsistencies between model size and performance. Our results demonstrate that OSS-Bench mitigates overfitting by leveraging the evolving complexity of OSS and highlights LLMs' limited understanding of low-level code security via extended fuzzing experiments. Overall, OSS-Bench offers a practical and scalable framework for benchmarking the real-world coding capabilities of LLMs.
Paper Structure (10 sections, 2 equations, 5 figures, 1 table)

This paper contains 10 sections, 2 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Approach
  3. Evaluation
  4. Evaluation Setup
  5. Effectiveness of OSS-Bench
  6. Fairness of OSS-Bench
  7. Extended Memory-Safety Evaluation via Fuzzing
  8. Discussion
  9. Related Work
  10. Conclusion

Figures (5)

  • Figure 1: Benchmark Workflow in OSS-Bench
  • Figure 2: OSS-Bench uses daily code prompts to simulate common LLM-assisted coding workflows.
  • Figure 3: Visualized Test Pass Rates of 1,000 Test Iterations in OSS-Bench$_{php}$
  • Figure 4: More memorization adds incorrect statements in advanced models
  • Figure 5: Models need caution when generating code for familiar content, such cases may not require optimization, but the presence of elements like lxb_encoding and iso-8859-2 thills LLMs.