Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models

Pareesa Ameneh Golnari, Adarsh Kumarappan, Wen Wen, Xiaoyu Liu, Gabriel Ryan, Yuting Sun, Shengyu Fu, Elsie Nallipogu

TL;DR

DevBench addresses the need for realistic, contamination-resistant code-generation evaluation by grounding tasks in observed developer telemetry and synthesizing 1,800 evaluation instances across six languages and six task categories. It adopts a multi-faceted evaluation combining functional correctness, semantic similarity, and LLM-based judgments to reveal strengths and gaps not captured by prior benchmarks. The study demonstrates variable model performance across languages and task types, with notable insights such as low-context pattern recognition outpacing complex NL-code translation and cross-language consistency. By open-sourcing the benchmark and evaluation pipeline, it enables targeted model improvement and practical tool selection for real-world development environments.

Abstract

DevBench is a telemetry-driven benchmark designed to evaluate Large Language Models (LLMs) on realistic code completion tasks. It includes 1,800 evaluation instances across six programming languages and six task categories derived from real developer telemetry, such as API usage and code purpose understanding. Unlike prior benchmarks, it emphasizes ecological validity, avoids training data contamination, and enables detailed diagnostics. The evaluation combines functional correctness, similarity-based metrics, and LLM-judge assessments focused on usefulness and contextual relevance. 9 state-of-the-art models were assessed, revealing differences in syntactic precision, semantic reasoning, and practical utility. Our benchmark provides actionable insights to guide model selection and improvement-detail that is often missing from other benchmarks but is essential for both practical deployment and targeted model development.

DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models

TL;DR

DevBench addresses the need for realistic, contamination-resistant code-generation evaluation by grounding tasks in observed developer telemetry and synthesizing 1,800 evaluation instances across six languages and six task categories. It adopts a multi-faceted evaluation combining functional correctness, semantic similarity, and LLM-based judgments to reveal strengths and gaps not captured by prior benchmarks. The study demonstrates variable model performance across languages and task types, with notable insights such as low-context pattern recognition outpacing complex NL-code translation and cross-language consistency. By open-sourcing the benchmark and evaluation pipeline, it enables targeted model improvement and practical tool selection for real-world development environments.

Abstract

DevBench is a telemetry-driven benchmark designed to evaluate Large Language Models (LLMs) on realistic code completion tasks. It includes 1,800 evaluation instances across six programming languages and six task categories derived from real developer telemetry, such as API usage and code purpose understanding. Unlike prior benchmarks, it emphasizes ecological validity, avoids training data contamination, and enables detailed diagnostics. The evaluation combines functional correctness, similarity-based metrics, and LLM-judge assessments focused on usefulness and contextual relevance. 9 state-of-the-art models were assessed, revealing differences in syntactic precision, semantic reasoning, and practical utility. Our benchmark provides actionable insights to guide model selection and improvement-detail that is often missing from other benchmarks but is essential for both practical deployment and targeted model development.
Paper Structure (42 sections, 1 equation, 3 figures, 9 tables)

This paper contains 42 sections, 1 equation, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Benchmark Design
  3. From User Telemetry to Categories
  4. Benchmark categories
  5. Benchmark construction
  6. Evaluation methods
  7. Functional correctness
  8. Similarity-Based evaluation
  9. LLM-judge evaluation
  10. Experiments
  11. Experimental setup
  12. Results and insights
  13. Functional correctness (Pass@1)
  14. Similarity-Based evaluation
  15. LLM-judge evaluation
  16. ...and 27 more sections

Figures (3)

  • Figure 1: This diagram presents the end-to-end DevBench pipeline, starting with developer telemetry analysis to define six code completion categories. Evaluation instances are synthetically generated and refined through human review. Final evaluation combines functional correctness, similarity-based metrics, and LLM-based judgment to assess functional, semantic, and holistic model performance.
  • Figure 2: Overall LLM-judge evaluation scores with 95% confidence intervals.
  • Figure 3: Breakdown of LLM-judge scores across models.