Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-itation is the Sincerest Form of Data: Generating Synthetic Buggy Code Submissions for Computing Education

Juho Leinonen, Paul Denny, Olli Kiljunen, Stephen MacNeil, Sami Sarsa, Arto Hellas

TL;DR

This work explores generating synthetic buggy code submissions for introductory programming exercises using GPT-4o and suggests that LLMs can be used to generate synthetic incorrect submissions that are not significantly different from real student data with regard to test case failure distributions.

Abstract

There is a great need for data in computing education research. Data is needed to understand how students behave, to train models of student behavior to optimally support students, and to develop and validate new assessment tools and learning analytics techniques. However, relatively few computing education datasets are shared openly, often due to privacy regulations and issues in making sure the data is anonymous. Large language models (LLMs) offer a promising approach to create large-scale, privacy-preserving synthetic data, which can be used to explore various aspects of student learning, develop and test educational technologies, and support research in areas where collecting real student data may be challenging or impractical. This work explores generating synthetic buggy code submissions for introductory programming exercises using GPT-4o. We compare the distribution of test case failures between synthetic and real student data from two courses to analyze the accuracy of the synthetic data in mimicking real student data. Our findings suggest that LLMs can be used to generate synthetic incorrect submissions that are not significantly different from real student data with regard to test case failure distributions. Our research contributes to the development of reliable synthetic datasets for computing education research and teaching, potentially accelerating progress in the field while preserving student privacy.

LLM-itation is the Sincerest Form of Data: Generating Synthetic Buggy Code Submissions for Computing Education

TL;DR

This work explores generating synthetic buggy code submissions for introductory programming exercises using GPT-4o and suggests that LLMs can be used to generate synthetic incorrect submissions that are not significantly different from real student data with regard to test case failure distributions.

Abstract

There is a great need for data in computing education research. Data is needed to understand how students behave, to train models of student behavior to optimally support students, and to develop and validate new assessment tools and learning analytics techniques. However, relatively few computing education datasets are shared openly, often due to privacy regulations and issues in making sure the data is anonymous. Large language models (LLMs) offer a promising approach to create large-scale, privacy-preserving synthetic data, which can be used to explore various aspects of student learning, develop and test educational technologies, and support research in areas where collecting real student data may be challenging or impractical. This work explores generating synthetic buggy code submissions for introductory programming exercises using GPT-4o. We compare the distribution of test case failures between synthetic and real student data from two courses to analyze the accuracy of the synthetic data in mimicking real student data. Our findings suggest that LLMs can be used to generate synthetic incorrect submissions that are not significantly different from real student data with regard to test case failure distributions. Our research contributes to the development of reliable synthetic datasets for computing education research and teaching, potentially accelerating progress in the field while preserving student privacy.

Paper Structure

This paper contains 14 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Bugs
  4. LLMs in Computing Education
  5. Generating Synthetic Data
  6. Methods
  7. Context and Data
  8. C Context
  9. Dart Context
  10. Prompting the LLM
  11. Analysis
  12. Results and Discussion
  13. Limitations
  14. Conclusions

Figures (1)

  • Figure 1: The prompts used in the study. The baseline prompt did not include the blue and yellow highlighted parts. The test-case-informed prompt included the blue highlighted part on top of the baseline prompt. The frequency-informed prompt included both the blue and yellow highlighted parts on top of the baseline prompt. The bolded parts indicate variables for which content depended on the exercise.