From Slides to Chatbots: Enhancing Large Language Models with University Course Materials

TL;DR

This paper tackles the challenge of enabling LLMs to answer university-level CS questions by leveraging course materials such as slides and transcripts. It compares Retrieval-Augmented Generation (RAG) and Continual Pre-Training (CPT) for domain adaptation and introduces a multimodal RAG variant that preserves slide visuals by feeding slide images to visually capable LLMs. The empirical results show that RAG outperforms CPT on the relatively small, multimodal course-material corpus, with slides-as-images providing additional gains over text-only retrieval. The findings offer practical, scalable strategies for building AI tutors in higher education and highlight the importance of preserving visual and structural information in educational resources.

Abstract

Large Language Models (LLMs) have advanced rapidly in recent years. One application of LLMs is to support student learning in educational settings. However, prior work has shown that LLMs still struggle to answer questions accurately within university-level computer science courses. In this work, we investigate how incorporating university course materials can enhance LLM performance in this setting. A key challenge lies in leveraging diverse course materials such as lecture slides and transcripts, which differ substantially from typical textual corpora: slides also contain visual elements like images and formulas, while transcripts contain spoken, less structured language. We compare two strategies, Retrieval-Augmented Generation (RAG) and Continual Pre-Training (CPT), to extend LLMs with course-specific knowledge. For lecture slides, we further explore a multi-modal RAG approach, where we present the retrieved content to the generator in image form. Our experiments reveal that, given the relatively small size of university course materials, RAG is more effective and efficient than CPT. Moreover, incorporating slides as images in the multi-modal setting significantly improves performance over text-only retrieval. These findings highlight practical strategies for developing AI assistants that better support learning and teaching, and we hope they inspire similar efforts in other educational contexts.

Figures (4)

• Figure 1: Retrieval-Augmented Generation (RAG) workflow. The retriever encodes slide text and lecture transcripts into a text-based vector database. At inference time, relevant text chunks or slides are retrieved. Left-hand side: text-only RAG. Right-hand side: multimodal RAG: when using a visually capable LLM, retrieved slides are passed as images to the generator, preserving visual information.
• Figure 2: CPT workflow, assuming the availability of both base and instruction-tuned versions of the LLM.
• Figure 3: Example of multimodal RAG effectiveness: Qwen2-VL produces a more accurate answer when provided with the slide as an image rather than as plain text. Viewing the slide image preserves the visual context, specifically, the arrows illustrating the temporal scope of usability and user experience.
• Figure 4: Course materials and Wikipedia (replay) validation perplexity for CPT with LLaMA 3.1 8B.