Integrating AI Tutors in a Programming Course

TL;DR

RAGMan presents a retrieval-augmented, conversation-centric approach to AI tutoring in an introductory programming course, deploying five homework-focused tutors to guide students without revealing solutions. By grounding responses in two knowledge bases and enforcing guardrails, the system achieves high accuracy within its scope and fosters a judgment-free learning environment, while enabling scalable, on-demand help for a large enrollment. The study reports favorable student perceptions and a statistically suggestive but non-causal improvement in WP grades, highlighting both potential benefits and the need for controlled evaluation to disentangle effects from cohort differences. Overall, the work demonstrates a practical pathway for integrating aligned AI tutors into CS education and motivates further research on pedagogy, evaluation design, and systems optimization.

Abstract

RAGMan is an LLM-powered tutoring system that can support a variety of course-specific and homework-specific AI tutors. RAGMan leverages Retrieval Augmented Generation (RAG), as well as strict instructions, to ensure the alignment of the AI tutors' responses. By using RAGMan's AI tutors, students receive assistance with their specific homework assignments without directly obtaining solutions, while also having the ability to ask general programming-related questions. RAGMan was deployed as an optional resource in an introductory programming course with an enrollment of 455 students. It was configured as a set of five homework-specific AI tutors. This paper describes the interactions the students had with the AI tutors, the students' feedback, and a comparative grade analysis. Overall, about half of the students engaged with the AI tutors, and the vast majority of the interactions were legitimate homework questions. When students posed questions within the intended scope, the AI tutors delivered accurate responses 98% of the time. Within the students used AI tutors, 78% reported that the tutors helped their learning. Beyond AI tutors' ability to provide valuable suggestions, students reported appreciating them for fostering a safe learning environment free from judgment.

Figures (6)

• Figure 1: RAGMan Architecture
• Figure 2: In-scope Questions
• Figure 3: Out-of-scope Questions
• Figure 4: In Scope Question, Good Response (sample: 12)
• Figure 5: In Scope Question, Bad Response (sample: 187)