Exploring Student Behaviors and Motivations using AI TAs with Optional Guardrails
Amanpreet Kapoor, Marc Diaz, Stephen MacNeil, Leo Porter, Paul Denny
TL;DR
This study examines how students interact with an AI teaching assistant that offers optional guardrails in a large programming course. Students could bypass safeguards using a See Solution feature to obtain full model outputs, enabling analysis of help-seeking behaviors and performance. The authors deployed Edugator with three coding tasks, collecting both quantitative interaction data and qualitative responses from 885 participants; about half used the See Solution feature at least once, and 14% used it on all tasks, with low-performing students more likely to seek solutions and procrastinate near deadlines. Findings highlight a tension between safeguarding pedagogy and student preferences, suggesting AI TAs should balance scaffolded guidance with flexible access to solutions to support learning while mitigating over-reliance.
Abstract
AI-powered chatbots and digital teaching assistants (AI TAs) are gaining popularity in programming education, offering students timely and personalized feedback. Despite their potential benefits, concerns about student over-reliance and academic misconduct have prompted the introduction of "guardrails" into AI TAs - features that provide scaffolded support rather than direct solutions. However, overly restrictive guardrails may lead students to bypass these tools and use unconstrained AI models, where interactions are not observable, thus limiting our understanding of students' help-seeking behaviors. To investigate this, we designed and deployed a novel AI TA tool with optional guardrails in one lab of a large introductory programming course. As students completed three code writing and debugging tasks, they had the option to receive guardrailed help or use a "See Solution" feature which disabled the guardrails and generated a verbatim response from the underlying model. We investigate students' motivations and use of this feature and examine the association between usage and their course performance. We found that 50% of the 885 students used the "See Solution" feature for at least one problem and 14% used it for all three problems. Additionally, low-performing students were more likely to use this feature and use it close to the deadline as they started assignments later. The predominant factors that motivated students to disable the guardrails were assistance in solving problems, time pressure, lack of self-regulation, and curiosity. Our work provides insights into students' solution-seeking motivations and behaviors, which has implications for the design of AI TAs that balance pedagogical goals with student preferences.