Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KASER: Knowledge-Aligned Student Error Simulator for Open-Ended Coding Tasks

Zhangqi Duan, Nigel Fernandez, Andrew Lan

TL;DR

KASER addresses the challenge of simulating open-ended student coding errors by aligning errors with interpretable knowledge components and training a code predictor via group-relative policy optimization (GRPO). It introduces a three-part reward—code similarity, error overlap, and code diversity—to mitigate mode collapse and produce diverse yet knowledge-consistent predictions, validated on CodeWorkout and FalconCode datasets. The approach combines a knowledge estimator from knowledge tracing with an RL-trained predictor and an automated error-annotation pipeline to enable fine-grained evaluation at both the per-student-problem and per-problem levels. Results show KASER outperforms strong baselines in code and error prediction and error coverage, with qualitative analyses illustrating how errors vary with knowledge profiles. The work has practical implications for educational diagnostics and personalized feedback, while acknowledging limitations in syntax-error generation and label reliability that warrant future improvement.

Abstract

Open-ended tasks, such as coding problems that are common in computer science education, provide detailed insights into student knowledge. However, training large language models (LLMs) to simulate and predict possible student errors in their responses to these problems can be challenging: they often suffer from mode collapse and fail to fully capture the diversity in syntax, style, and solution approach in student responses. In this work, we present KASER (Knowledge-Aligned Student Error Simulator), a novel approach that aligns errors with student knowledge. We propose a training method based on reinforcement learning using a hybrid reward that reflects three aspects of student code prediction: i) code similarity to the ground-truth, ii) error matching, and iii) code prediction diversity. On two real-world datasets, we perform two levels of evaluation and show that: At the per-student-problem pair level, our method outperforms baselines on code and error prediction; at the per-problem level, our method outperforms baselines on error coverage and simulated code diversity.

KASER: Knowledge-Aligned Student Error Simulator for Open-Ended Coding Tasks

TL;DR

KASER addresses the challenge of simulating open-ended student coding errors by aligning errors with interpretable knowledge components and training a code predictor via group-relative policy optimization (GRPO). It introduces a three-part reward—code similarity, error overlap, and code diversity—to mitigate mode collapse and produce diverse yet knowledge-consistent predictions, validated on CodeWorkout and FalconCode datasets. The approach combines a knowledge estimator from knowledge tracing with an RL-trained predictor and an automated error-annotation pipeline to enable fine-grained evaluation at both the per-student-problem and per-problem levels. Results show KASER outperforms strong baselines in code and error prediction and error coverage, with qualitative analyses illustrating how errors vary with knowledge profiles. The work has practical implications for educational diagnostics and personalized feedback, while acknowledging limitations in syntax-error generation and label reliability that warrant future improvement.

Abstract

Open-ended tasks, such as coding problems that are common in computer science education, provide detailed insights into student knowledge. However, training large language models (LLMs) to simulate and predict possible student errors in their responses to these problems can be challenging: they often suffer from mode collapse and fail to fully capture the diversity in syntax, style, and solution approach in student responses. In this work, we present KASER (Knowledge-Aligned Student Error Simulator), a novel approach that aligns errors with student knowledge. We propose a training method based on reinforcement learning using a hybrid reward that reflects three aspects of student code prediction: i) code similarity to the ground-truth, ii) error matching, and iii) code prediction diversity. On two real-world datasets, we perform two levels of evaluation and show that: At the per-student-problem pair level, our method outperforms baselines on code and error prediction; at the per-problem level, our method outperforms baselines on error coverage and simulated code diversity.
Paper Structure (44 sections, 6 equations, 2 figures, 13 tables)

This paper contains 44 sections, 6 equations, 2 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Task Setup
  4. Error Annotation
  5. Error Generation
  6. Clustering Errors
  7. Representative Labels for Error Clusters
  8. Student Error Simulation
  9. Methodology
  10. Student Knowledge Estimator
  11. Student Code Predictor
  12. Knowledge-Aligned Student Error Simulation via RL
  13. Reward Design for RL Training
  14. Code Similarity Reward
  15. Error Match Reward
  16. ...and 29 more sections

Figures (2)

  • Figure 1: KASER estimates an interpretable student knowledge profile and trains an LLM via GRPO with a hybrid reward to simulate knowledge-aligned errors in predicted student code.
  • Figure 2: Diversity visualization of student codes. KASER predicts student codes with higher diversity and alignment with ground-truth student codes compared to StudentSFT.