Student Engagement with GenAI's Tutoring Feedback: A Mixed Methods Study
Sven Jacobs, Jan Haas, Natalie Kiesler
TL;DR
This study investigates how undergraduate programmers engage with GenAI-generated tutoring feedback during Python tasks using a mixed-methods design that combines think-aloud protocols and eye-tracking in the Tutor Kai environment. It identifies four themes in students' verbalized thoughts—understanding, need for more information, disagreement, and explicit judgments—and maps these to their immediate actions, revealing that about 82% of GenAI feedback requests lead to at least one improvement. Despite high reading rates, students often collect additional feedback or refrain from immediate modification when they disagree or require further details, highlighting the role of internal evaluation and feedback literacy in effective engagement. The findings inform tool developers and educators about designing more flexible, interpretable GenAI tutoring interfaces and pedagogical strategies to support self-regulated, goal-directed feedback use in programming education.
Abstract
How students utilize immediate tutoring feedback in programming education depends on various factors. Among them are the feedback quality, but also students' engagement, i.e., their perception, interpretation, and use of feedback. However, there is limited research on how students engage with various types of tutoring feedback. For this reason, we developed a learning environment that provides students with Python programming tasks and various types of immediate, AI-generated tutoring feedback. The feedback is displayed within four components. Using a mixed-methods approach (think-aloud study and eye-tracking), we conducted a study with 20 undergraduate students enrolled in an introductory programming course. Our research aims to: (1) identify what students think when they engage with the tutoring feedback components, and (2) explore the relations between the tutoring feedback components, students' visual attention, verbalized thoughts, and their immediate actions as part of the problem-solving process. The analysis of students' thoughts while engaging with 380 feedback components revealed four main themes: students express understanding or disagreement, additional information needed, and students explicitly judge the feedback. Exploring the relations between feedback, students' attention, thoughts, and actions showed a clear relationship. While expressions of understanding were associated with improvements, expressions of disagreement or need for additional information prompted students to collect another feedback component rather than act on the current information. These insights into students' engagement and decision-making processes contribute to an increased understanding of tutoring feedback and how students engage with it. Thereby, this work has implications for tool developers and educators facilitating feedback.