Prediction of Item Difficulty for Reading Comprehension Items by Creation of Annotated Item Repository

TL;DR

The paper tackles predicting reading comprehension item difficulty by converting p-values from NY and TX state tests into a common IRT-like scale and predicting this difficulty from a rich annotated item repository. It combines linguistic, test, and context features with LLM embeddings (BERT, ModernBERT, LlAMA) in a penalized regression framework, achieving RMSE around 0.59–0.62 and correlations up to 0.77 on test data. The results show that either text-based features or LLM embeddings can yield strong predictions, with the best performance arising from combining all features and applying PCA to embeddings to suit the modest dataset size. This work enables early item screening and filtering for reading assessments and provides publicly available data resources for researchers and practitioners, while highlighting avenues for domain-specific tuning and broader data inclusion in future work.

Abstract

Prediction of item difficulty based on its text content is of substantial interest. In this paper, we focus on the related problem of recovering IRT-based difficulty when the data originally reported item p-value (percent correct responses). We model this item difficulty using a repository of reading passages and student data from US standardized tests from New York and Texas for grades 3-8 spanning the years 2017-23. This repository is annotated with meta-data on (1) linguistic features of the reading items, (2) test features of the passage, and (3) context features. A penalized regression prediction model with all these features can predict item difficulty with RMSE 0.52 compared to baseline RMSE of 0.92, and with a correlation of 0.77 between true and predicted difficulty. We supplement these features with embeddings from LLMs (ModernBERT, BERT, and LlAMA), which marginally improve item difficulty prediction. When models use only item linguistic features or LLM embeddings, prediction performance is similar, which suggests that only one of these feature categories may be required. This item difficulty prediction model can be used to filter and categorize reading items and will be made publicly available for use by other stakeholders.

Figures (3)

• Figure 1: Conversion of p-value to IRT Easiness by Grades
• Figure 2: Predictors for Item Difficulty Model: Input text to generate embeddings, and Linguistic, Test, and Context Features
• Figure B.1: Robustness to Vertical Scale Choice