Automated Alignment of Math Items to Content Standards in Large-Scale Assessments Using Language Models

TL;DR

This study addresses automated alignment of math items to content standards in large-scale assessments by comparing embedding-based, transformer-based, and ensemble approaches. Using a dataset of 1,385 items annotated with 4 domains and 19 skills, it shows that DeBERTa-v3-base achieves the best domain alignment (F1 ≈ 0.950) and RoBERTa-large the best skill alignment (F1 ≈ 0.869), while ensemble methods offer limited improvements over the top single models. Embedding-based methods benefited significantly from PCA for linear classifiers but underperformed deep transformers, highlighting a trade-off between dimensionality reduction and nonlinear modeling. The findings inform practical model selection for scalable, accurate item alignment and point to future work in efficient fine-tuning, diversified ensembles, and domain-adapted pretraining to balance accuracy with resource constraints.

Abstract

Accurate alignment of items to content standards is critical for valid score interpretation in large-scale assessments. This study evaluates three automated paradigms for aligning items with four domain and nineteen skill labels. First, we extracted embeddings and trained multiple classical supervised machine learning models, and further investigated the impact of dimensionality reduction on model performance. Second, we fine-tuned eight BERT model and its variants for both domain and skill alignment. Third, we explored ensemble learning with majority voting and stacking with multiple meta-models. The DeBERTa-v3-base achieved the highest weighted-average F1 score of 0.950 for domain alignment while the RoBERTa-large yielded the highest F1 score of 0.869 for skill alignment. Ensemble models did not surpass the best-performing language models. Dimension reduction enhanced linear classifiers based on embeddings but did not perform better than language models. This study demonstrated different methods in automated item alignment to content standards.}