PS$^2$: Parameterized Control for Fine-Grained Student Proficiency Simulation

TL;DR

PS2 addresses the challenge of simulating student proficiency with fine granularity in data-scarce settings by parameterizing proficiency as a logit-level interpolation between an upper-bound LLM and a cognitively informed lower-bound LLM. A cognitive error generation strategy provides structured, plausible low-proficiency behavior, and calibration aligns simulated proficiency with target performance, enabling continuous and calibrated simulations. Across two public datasets, PS2 outperforms prompt-based baselines in both question-difficulty prediction and distributional alignment of simulated responses, with ablations confirming the value of the lower-bound training. The work offers a practical, scalable approach for evaluating instructional content and interventions without real student data, with code available for replication.

Abstract

Understanding how students with different proficiency levels respond to educational materials is a critical issue within the field of AI for Education. However, acquiring sufficient real student response data for a robust evaluation is often hindered by cost, ethics, and security constraints. Consequently, LLM-based student proficiency simulation, especially prompt-based methods, has emerged as a practical alternative under data-scarce conditions. Despite their promise, current methods still exhibit limited controllability with coarse-grained proficiency representations, high sensitivity to prompt design, and the lack of calibration with academic performance. Therefore, we propose Parameterized Student Proficiency Simulation (PS$^2$), an unsupervised and parameterized model-level framework that simulates students with different proficiencies by interpolating between a strong upper-bound LLM and a weaker, cognitive error-informed lower-bound student LLM via a hybrid ratio. Specifically, the lower-bound model is constructed by fine-tuning the weaker LM to exhibit cognitive errors when responding to educational materials. To ensure alignment with target proficiency levels, PS$^2$ further calibrates the interpolation ratio with academic performance. Experiments on two public datasets demonstrate that PS$^2$ achieves finer-grained and consistent proficiency simulation compared to existing baselines, leading to superior performance in student behavior similarity and item difficulty prediction.

Figures (4)

• Figure 1: Prompt-based Student Proficiency Simulation v.s. Parameterized Student Proficiency Simulation.
• Figure 2: The top panel shows PS2 simulating students by hybridizing upper- and lower-bound LLMs at the logits level, while the bottom panels depict lower-bound training with cognitive errors and calibration of hybrid ratios using assessment scores.
• Figure 3: Accuracy of student answers under 3/5/7 proficiency levels in Eedi (top-left: Condition Prompt LLM; top-right: Knowledge Prompt LLM; bottom-left: PS2). Bottom-right reports SCC for question difficulty prediction.
• Figure 4: Scores (normalized) of different low-bound model variants across datasets.