Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Educational Program Repair

Charles Koutcheme, Nicola Dainese, Sami Sarsa, Juho Leinonen, Arto Hellas, Paul Denny

TL;DR

This work addresses the lack of standardized benchmarks for educational program repair by formalizing the task and introducing a unified evaluation framework with a novel rouge@k metric. It curates two public datasets, FalconCode and Singapore, and provides baseline results for five decoder-only models, examining how performance scales with compute, model size, and cross-dataset transfer. The contributions enable reproducible, fair comparisons across methods and lay groundwork for broader educational feedback applications, including potential extensions to explanations and multi-language support. The benchmark aims to accelerate progress in educational AI by providing a reference point for evaluating repair quality and student-facing feedback mechanisms.

Abstract

The emergence of large language models (LLMs) has sparked enormous interest due to their potential application across a range of educational tasks. For example, recent work in programming education has used LLMs to generate learning resources, improve error messages, and provide feedback on code. However, one factor that limits progress within the field is that much of the research uses bespoke datasets and different evaluation metrics, making direct comparisons between results unreliable. Thus, there is a pressing need for standardization and benchmarks that facilitate the equitable comparison of competing approaches. One task where LLMs show great promise is program repair, which can be used to provide debugging support and next-step hints to students. In this article, we propose a novel educational program repair benchmark. We curate two high-quality publicly available programming datasets, present a unified evaluation procedure introducing a novel evaluation metric rouge@k for approximating the quality of repairs, and evaluate a set of five recent models to establish baseline performance.

Benchmarking Educational Program Repair

TL;DR

This work addresses the lack of standardized benchmarks for educational program repair by formalizing the task and introducing a unified evaluation framework with a novel rouge@k metric. It curates two public datasets, FalconCode and Singapore, and provides baseline results for five decoder-only models, examining how performance scales with compute, model size, and cross-dataset transfer. The contributions enable reproducible, fair comparisons across methods and lay groundwork for broader educational feedback applications, including potential extensions to explanations and multi-language support. The benchmark aims to accelerate progress in educational AI by providing a reference point for evaluating repair quality and student-facing feedback mechanisms.

Abstract

The emergence of large language models (LLMs) has sparked enormous interest due to their potential application across a range of educational tasks. For example, recent work in programming education has used LLMs to generate learning resources, improve error messages, and provide feedback on code. However, one factor that limits progress within the field is that much of the research uses bespoke datasets and different evaluation metrics, making direct comparisons between results unreliable. Thus, there is a pressing need for standardization and benchmarks that facilitate the equitable comparison of competing approaches. One task where LLMs show great promise is program repair, which can be used to provide debugging support and next-step hints to students. In this article, we propose a novel educational program repair benchmark. We curate two high-quality publicly available programming datasets, present a unified evaluation procedure introducing a novel evaluation metric rouge@k for approximating the quality of repairs, and evaluate a set of five recent models to establish baseline performance.
Paper Structure (29 sections, 2 figures, 3 tables)

This paper contains 29 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmarking program synthesis.
  4. Benchmarking program repair.
  5. Educational Program Repair
  6. Task and Evaluation Procedure
  7. Problem setup and motivations.
  8. Evaluation procedure.
  9. pass@k
  10. rouge@k
  11. Educational Scenarios and Datasets
  12. Prior data available.
  13. No prior data available.
  14. Baseline and Experiments
  15. Models
  16. ...and 14 more sections

Figures (2)

  • Figure 1: Overview of the program repair task and its evaluation procedure. (1) For each incorrect program, we pass to a language model a prompt which contains the programming task, and the incorrect student code. In addition, the prompt may contain more contextual information, e.g. grader's feedback. (2) The model generates multiple feedbacks based on the instructions; each feedback must contain (at least) a full correction to the incorrect program. (3) We extract the full code repairs, which we (4) pass to unit tests. The samples which pass or fail the unit tests allow us to evaluate (5) functional correctness and (6) closeness to the incorrect program.
  • Figure 2: The prompt template used for model training and evaluation: (1) a preamble orienting the model to the task of the program repair (although pretrained models are not instruction tuned, a preamble remains a useful signal), (2) the problem description extracted from the dataset, (3) an indication of the start of the buggy program, (4) the buggy program to fix, (5) an indication of the start of the repaired program, (6) the artificial repair found for the buggy program (see section \ref{['sec:app-art']}) -- included only in the training phase.