Leveraging Large Language Models to Generate Course-specific Semantically Annotated Learning Objects
Dominic Lohr, Marc Berges, Abhishek Chugh, Michael Kohlhase, Dennis Müller
TL;DR
The paper addresses generating university-level CS learning content that is both course-contextual and semantically annotated for automatic learner modeling. It deploys a retrieval-augmented generation pipeline with a large language model and semantic annotations via sTeX/OMDoc to produce autogradable questions and rich annotations. Results show robust generation of structural semantic annotations and course-aligned questions, but relational annotations and high-quality educational feedback remain challenging, requiring substantial human curation. The work demonstrates the promise of AI-assisted content creation for adaptive learning while identifying key limitations and outlining concrete paths for improving annotation quality and automation in future work.
Abstract
Background: Over the past few decades, the process and methodology of automated question generation (AQG) have undergone significant transformations. Recent progress in generative natural language models has opened up new potential in the generation of educational content. Objectives: This paper explores the potential of large language models (LLMs) for generating computer science questions that are sufficiently annotated for automatic learner model updates, are fully situated in the context of a particular course, and address the cognitive dimension understand. Methods: Unlike previous attempts that might use basic methods like ChatGPT, our approach involves more targeted strategies such as retrieval-augmented generation (RAG) to produce contextually relevant and pedagogically meaningful learning objects. Results and Conclusions: Our results show that generating structural, semantic annotations works well. However, this success was not reflected in the case of relational annotations. The quality of the generated questions often did not meet educational standards, highlighting that although LLMs can contribute to the pool of learning materials, their current level of performance requires significant human intervention to refine and validate the generated content.