Leveraging Large Language Models for Identifying Knowledge Components
Canwen Wang, Jionghao Lin, Kenneth R. Koedinger
TL;DR
This paper tackles the KC identification bottleneck in adaptive learning by scaling a simulated textbook prompting approach to 646 MCQs and comparing it against an expert-designed KC model. The initial LLM-driven KC generation yields $RMSE = 0.4285$ and 569 KCs, highlighting substantial label redundancy. Introducing a cosine similarity-based merging of KC labels, using OpenAI embeddings, reduces the KC count to 428 with an improved RMSE of $0.4259$ at a threshold of 0.8, approaching the expert model's performance. The results show that combining scaled LLM generation with semantic merging offers a viable path to automated, refined KC identification, with implications for scalable adaptive learning systems.
Abstract
Knowledge Components (KCs) are foundational to adaptive learning systems, but their manual identification by domain experts is a significant bottleneck. While Large Language Models (LLMs) offer a promising avenue for automating this process, prior research has been limited to small datasets and has been shown to produce superfluous, redundant KC labels. This study addresses these limitations by first scaling a "simulated textbook" LLM prompting strategy (using GPT-4o-mini) to a larger dataset of 646 multiple-choice questions. We found that this initial automated approach performed significantly worse than an expert-designed KC model (RMSE 0.4285 vs. 0.4206) and generated an excessive number of KCs (569 vs. 101). To address the issue of redundancy, we proposed and evaluated a novel method for merging semantically similar KC labels based on their cosine similarity. This merging strategy significantly improved the model's performance; a model using a cosine similarity threshold of 0.8 achieved the best result, reducing the KC count to 428 and improving the RMSE to 0.4259. This demonstrates that while scaled LLM generation alone is insufficient, combining it with a semantic merging technique offers a viable path toward automating and refining KC identification.