Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synthetic Student Responses: LLM-Extracted Features for IRT Difficulty Parameter Estimation

Matias Hoyl

TL;DR

This work tackles the resource-intensive problem of estimating IRT difficulty without direct student testing by modeling the response process and incorporating LLM-derived pedagogical features. A two-stage framework first predicts correctness for student-question pairs using a neural network that fuses question embeddings, linguistic and structural features, option characteristics, LLM-extracted signals, and student embeddings, then derives $1$-parameter logistic ($1$-PL) IRT difficulties from the simulated responses. The approach achieves a strong alignment with benchmark difficulties ($r$ ≈ $0.78$, $\rho$ ≈ $0.76$, RMSE ≈ $1.19$) on unseen items and demonstrates substantial data-efficiency: the NN-based difficulties equate to about $5{,}818$ real answers (≈ $22.9\%$) for the same accuracy. Ablation shows LLM-derived features provide the largest gains, underscoring the value of pedagogical signals such as solution steps, cognitive demand, and mis_conceptions in estimating item difficulty without actual testing, enabling faster, cost-effective assessment development with maintained psychometric validity.

Abstract

Educational assessment relies heavily on knowing question difficulty, traditionally determined through resource-intensive pre-testing with students. This creates significant barriers for both classroom teachers and assessment developers. We investigate whether Item Response Theory (IRT) difficulty parameters can be accurately estimated without student testing by modeling the response process and explore the relative contribution of different feature types to prediction accuracy. Our approach combines traditional linguistic features with pedagogical insights extracted using Large Language Models (LLMs), including solution step count, cognitive complexity, and potential misconceptions. We implement a two-stage process: first training a neural network to predict how students would respond to questions, then deriving difficulty parameters from these simulated response patterns. Using a dataset of over 250,000 student responses to mathematics questions, our model achieves a Pearson correlation of approximately 0.78 between predicted and actual difficulty parameters on completely unseen questions.

Synthetic Student Responses: LLM-Extracted Features for IRT Difficulty Parameter Estimation

TL;DR

This work tackles the resource-intensive problem of estimating IRT difficulty without direct student testing by modeling the response process and incorporating LLM-derived pedagogical features. A two-stage framework first predicts correctness for student-question pairs using a neural network that fuses question embeddings, linguistic and structural features, option characteristics, LLM-extracted signals, and student embeddings, then derives -parameter logistic (-PL) IRT difficulties from the simulated responses. The approach achieves a strong alignment with benchmark difficulties ( ≈ , ≈ , RMSE ≈ ) on unseen items and demonstrates substantial data-efficiency: the NN-based difficulties equate to about real answers (≈ ) for the same accuracy. Ablation shows LLM-derived features provide the largest gains, underscoring the value of pedagogical signals such as solution steps, cognitive demand, and mis_conceptions in estimating item difficulty without actual testing, enabling faster, cost-effective assessment development with maintained psychometric validity.

Abstract

Educational assessment relies heavily on knowing question difficulty, traditionally determined through resource-intensive pre-testing with students. This creates significant barriers for both classroom teachers and assessment developers. We investigate whether Item Response Theory (IRT) difficulty parameters can be accurately estimated without student testing by modeling the response process and explore the relative contribution of different feature types to prediction accuracy. Our approach combines traditional linguistic features with pedagogical insights extracted using Large Language Models (LLMs), including solution step count, cognitive complexity, and potential misconceptions. We implement a two-stage process: first training a neural network to predict how students would respond to questions, then deriving difficulty parameters from these simulated response patterns. Using a dataset of over 250,000 student responses to mathematics questions, our model achieves a Pearson correlation of approximately 0.78 between predicted and actual difficulty parameters on completely unseen questions.
Paper Structure (56 sections, 1 equation, 7 figures, 6 tables)

This paper contains 56 sections, 1 equation, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional Item Difficulty Estimation
  4. Text-based Features for Difficulty Prediction
  5. Linguistic Features
  6. Domain-specific Features
  7. Machine Learning Approaches for Difficulty Prediction
  8. Classical Machine Learning Methods
  9. Neural Network Models
  10. Transformer-based Approaches
  11. Feature Extraction Using Language Models
  12. Gaps in Current Research
  13. Data
  14. Methods
  15. Stage 1: Neural Network for Correctness Prediction
  16. ...and 41 more sections

Figures (7)

  • Figure 1: Data processing pipeline.
  • Figure 2: Stage 1: Generation of a correctness prediction matrix using a neural network.
  • Figure 3: Stage 2: Estimation of IRT difficulty parameters from a correctness prediction matrix.
  • Figure 4: Schematic of the neural network model for student correctness prediction, showing the integration of question embeddings, feature vectors, and student representations.
  • Figure 5: Correlation between benchmark 1PL IRT difficulty and NN-derived difficulty on the holdout test set.
  • ...and 2 more figures