RubberDuckBench: A Benchmark for AI Coding Assistants

TL;DR

This work tackles the gap in evaluating AI coding assistants on contextualized, project-specific code questions. It introduces RubberDuckBench, a multilingual benchmark of 15 real-world questions sourced from GitHub PR comments, accompanied by rubrics and a reproducible evaluation package. An extensive 20-model evaluation reveals that even top-performing systems struggle to deliver fully correct, non-hallucinated answers across all items, with substantial variation across languages and question types and no clear cost-performance advantage. The benchmark aims to catalyze future research toward trustworthy, accurate coding assistants by providing a concrete, repeatable evaluation framework and highlighting current limitations.

Abstract

Programmers are turning to AI coding assistants to answer questions about their code. Benchmarks are needed to soundly evaluate these systems and understand their performance. To enable such a study, we curate a benchmark of real-world contextualized questions derived from Github pull request comments. Out of this work, we present RubberDuckBench: a multilingual benchmark of questions about code, along with detailed rubrics for evaluating answers. We evaluate a diverse set of 20 LLMs (proprietary & open-source) on answering these questions. We find that even state of the art models fail to give consistent, correct responses across the benchmark. Grok 4 (69.29%), Claude Opus 4 (68.5%), and GPT-5 (67.8%) perform best overall, but do not exhibit pairwise significant superiority over the next 9 best performing models. Most models obtain points through partial credit, with the best performing models only answering at most 2 questions completely correctly across all trials. Furthermore, models often hallucinate with lies in 58.3\% of responses on average. Cost analysis reveals no correlation between expense (API pricing or parameter count) and performance. We intend this benchmark to be a target for future research in trustworthy and correct AI coding assistants.

Figures (5)

• Figure 1: PR comment exemplifying a contextualized question.
• Figure 3: Rubric: Difference Between Contextualized at vs [] Operators
• Figure 4: RQ1: Analysis of Model Performance
• Figure 5: RQ2: Resource Usage of Proprietary and Open Source Models.
• Figure 6: Point Deduction Per Error Type.