gencat: Generative computerized adaptive testing
Wanyong Feng, Andrew Lan
TL;DR
GENCAT tackles the limitation of traditional CAT by leveraging open-ended text and code through a Generative Item Response Theory (GIRT) framework, enabling LLM-driven knowledge estimation and diverse, information-rich question selection. It integrates SFT and Direct Preference Optimization to align KC mastery with generated responses, and introduces three selection strategies—Uncertainty, Diversity, and Information—based on the generative outputs. Across two real-world programming datasets, GENCAT yields higher accuracy and AUC in early testing stages (with improvements up to about 4.32 percentage points) and demonstrates robust test security and informative generated responses. The work advances adaptive testing with open-ended tasks, offering practical gains for short-test efficiency and detailed diagnostic insights, while outlining pathways for broader domain generalization and KC discovery.
Abstract
Existing computerized Adaptive Testing (CAT) frameworks are typically built on predicting the correctness of a student response to a question. Although effective, this approach fails to leverage textual information in questions and responses, especially for open-ended questions. In this work, we propose GENCAT (\textbf{GEN}erative \textbf{CAT}), a novel CAT framework that leverages Large Language Models for knowledge estimate and question selection. First, we develop a Generative Item Response Theory (GIRT) model that enables us to estimate student knowledge from their open-ended responses and predict responses to unseen questions. We train the model in a two-step process, first via Supervised Fine-Tuning and then via preference optimization for knowledge-response alignment. Second, we introduce three question selection algorithms that leverage the generative capabilities of the GIRT model, based on the uncertainty, linguistic diversity, and information of sampled student responses. Third, we conduct experiments on two real-world programming datasets and demonstrate that GENCAT outperforms existing CAT baselines, achieving an AUC improvement of up to 4.32\% in the key early testing stages.