Revolutionizing Undergraduate Learning: CourseGPT and Its Generative AI Advancements

TL;DR

This work tackles the challenge of integrating generative AI into undergraduate education by developing CourseGPT, a course-specific AI assistant built on open-source Mistral LLMs and a Retrieval Augmented Generation pipeline. It evaluates performance on the CPR E 431course using multiple metrics—correctness, context recall, and faithfulness—comparing Mistral-7b and Mixtral-8x7b, with Mixtral-8x7b achieving the highest correctness ($88.0\%$) and faithfulness ($66.6\%$). Results, together with student and TA feedback, indicate that larger models improve information accuracy and alignment with course context, while also enhancing instructor efficiency and enabling personalized learning experiences. The findings demonstrate the practicality of domain-specific AI assistants in higher education and suggest a path toward broader adoption across disciplines, with ongoing work addressing computational efficiency, scalability, and data relevance.

Abstract

Integrating Generative AI (GenAI) into educational contexts presents a transformative potential for enhancing learning experiences. This paper introduces CourseGPT, a generative AI tool designed to support instructors and enhance the educational experiences of undergraduate students. Built on open-source Large Language Models (LLMs) from Mistral AI, CourseGPT offers continuous instructor support and regular updates to course materials, enriching the learning environment. By utilizing course-specific content, such as slide decks and supplementary readings and references, CourseGPT provides precise, dynamically generated responses to student inquiries. Unlike generic AI models, CourseGPT allows instructors to manage and control the responses, thus extending the course scope without overwhelming details. The paper demonstrates the application of CourseGPT using the CPR E 431 - Basics of Information System Security course as a pilot. This course, with its large enrollments and diverse curriculum, serves as an ideal testbed for CourseGPT. The tool aims to enhance the learning experience, accelerate feedback processes, and streamline administrative tasks. The study evaluates CourseGPT's impact on student outcomes, focusing on correctness scores, context recall, and faithfulness of responses. Results indicate that the Mixtral-8x7b model, with a higher parameter count, outperforms smaller models, achieving an 88.0% correctness score and a 66.6% faithfulness score. Additionally, feedback from former students and teaching assistants on CourseGPT's accuracy, helpfulness, and overall performance was collected. The outcomes revealed that a significant majority found CourseGPT to be highly accurate and beneficial in addressing their queries, with many praising its ability to provide timely and relevant information.

Figures (7)

• Figure 1: CourseGPT Workflow
• Figure 2: RAG-LLM correctness scores distribution
• Figure 3: RAG-LLM performance metrics
• Figure 4: Helpfulness
• Figure 5: Accuracy