Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models

TL;DR

The paper addresses the cost and practicality of calibrating Item Response Theory parameters by proposing a method that implicitly models item difficulty and discrimination through fine-tuned large language models. By training Qwen-3 dense models with LoRA to simulate student responses conditioned on discrete ability descriptors, it reconstructs synthetic ICCs and maps them to continuous IRT parameters, evaluated on Grade 6 ELA data and the BEA 2024 Shared Task. The results show that LLM-based simulations can closely follow calibration curves, with larger models (notably Qwen-14B) achieving the strongest correlations for difficulty and discrimination, suggesting potential for low-stakes pre-testing and discrimination modeling. However, the approach relies on small datasets, prompts for ability descriptors, and careful validation before broader deployment in high-stakes assessments. Future work may include model-generated rationales, larger diverse datasets, and further refinement of distribution-correction and prompting strategies to improve reliability.

Abstract

Traditional methods for determining assessment item parameters, such as difficulty and discrimination, rely heavily on expensive field testing to collect student performance data for Item Response Theory (IRT) calibration. This study introduces a novel approach that implicitly models these psychometric properties by fine-tuning Large Language Models (LLMs) to simulate student responses across a spectrum of latent abilities. Leveraging the Qwen-3 dense model series and Low-Rank Adaptation (LoRA), we train models to generate responses to multiple choice questions conditioned on discrete ability descriptors. We reconstruct the probability of a correct response as a function of student ability, effectively generating synthetic Item Characteristic Curves (ICCs) to estimate IRT parameters. Evaluation on a dataset of Grade 6 English Language Arts (ELA) items and the BEA 2024 Shared Task dataset demonstrates that this method competes with or outperforms baseline approaches. This simulation-based technique seems particularly effective at modeling item discrimination.

Figures (3)

• Figure 1: Architectural Schematic of the Qwen3 Dense Model Series. This diagram illustrates the transformer block structure used across the Qwen3 lineup, highlighting the implementation of Root Mean Square Layer Normalization (RMSNorm) and Masked Grouped-query Attention. The architecture features a "pre-norm" configuration where RMSNorm 1 and RMSNorm 2 precede the attention and feed-forward layers, respectively. Key technical enhancements include Rotary Positional Embeddings (RoPE) and QKNorm to improve training stability.
• Figure 2: Distribution of Ability-Level Descriptors. The bar chart illustrates the frequency of student responses categorized by the 20 discrete descriptors, ranging from "Critical" to "Exemplary". The resulting distribution approximates a normal curve, consistent with the calibrated student ability parameters $\theta \sim \mathcal{N}(0.13, 1.15)$ observed in the Grade 6 English Language and Arts assessment data.
• Figure 3: Comparison of Item Characteristic Curves (ICCs). This plot illustrates the relationship between student ability levels ($\theta$) and the probability of a correct response ($P$) for a sample item. The blue curve represents the Observed Probabilities from empirical student data, while the green curve depicts the Calibration Probabilities derived from fitting the Nominal Response Model (NRM). The red curve shows the LLM Probabilities, demonstrating the language model's ability to simulate responses that adhere to specific ability level descriptors after supervised fine-tuning and distribution correction.