Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations

TL;DR

This work investigates whether open-source LLMs can reliably estimate the real difficulty of math test items by simulating student responses and fitting Item Response Theory (IRT) models to the simulated data. Unlike direct prompt-based predictions, the simulation pipeline—employing role-played students across NAEP-grade skill levels and diverse identity cues—yields correlations up to $r=0.82$ (Grade 12) with actual NAEP item difficulties, and strong AUCs ($0.78$–$0.90$) for discriminating hard vs easy items. A key finding is that smaller-to-moderate LLMs with weaker math abilities can outperform stronger models in producing realistic error patterns, and that diverse identities in prompts improve alignment with real-world performance. The approach demonstrates a cost-effective, scalable pre-screening tool for item development, though it entails notable limitations in distractor prediction, dataset scope, and demographic coverage, warranting further work to broaden applicability and address ethical considerations.

Abstract

Standardized math assessments require expensive human pilot studies to establish the difficulty of test items. We investigate the predictive value of open-source large language models (LLMs) for evaluating the difficulty of multiple-choice math questions for real-world students. We show that, while LLMs are poor direct judges of problem difficulty, simulation-based approaches with LLMs yield promising results under the right conditions. Under the proposed approach, we simulate a "classroom" of 4th, 8th, or 12th grade students by prompting the LLM to role-play students of varying proficiency levels. We use the outcomes of these simulations to fit Item Response Theory (IRT) models, comparing learned difficulty parameters for items to their real-world difficulties, as determined by item-level statistics furnished by the National Assessment of Educational Progress (NAEP). We observe correlations as high as 0.75, 0.76, and 0.82 for grades 4, 8, and 12, respectively. In our simulations, we experiment with different "classroom sizes," showing tradeoffs between computation size and accuracy. We find that role-plays with named students improves predictions (compared to student ids), and stratifying names across gender and race further improves predictions. Our results show that LLMs with relatively weaker mathematical abilities (Gemma) actually yield better real-world difficulty predictions than mathematically stronger models (Llama and Qwen), further underscoring the suitability of open-source models for the task.

Figures (9)

• Figure 1: We simulate classrooms by prompting LLMs at different skill levels. Simulated students with varying abilities respond to question items . An IRT model is fit to the simulated responses to estimate students' ability, $\beta$ and item difficulty, $\delta$.
• Figure 2: IRT group ability estimates for for different LLMs across different skill levels.
• Figure 3: Distribution of content areas being tested in the dataset.
• Figure 4: Average Percentage Correct Score by NAEP Assigned Difficulty at a grade and content area breakdown.
• Figure 5: Average Accuracy rates for simulated students of different skill levels across grades, 4, 8 and 12.