Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Closing the Loop: An Instructor-in-the-Loop AI Assistance System for Supporting Student Help-Seeking in Programming Education

Tung Phung, Heeryung Choi, Mengyan Wu, Christopher Brooks, Sumit Gulwani, Adish Singla

TL;DR

The paper tackles the challenge of providing timely, high-quality feedback in programming education at scale. It introduces a hybrid instructor-in-the-loop system that delivers AI-generated hints and escalates to human instructors when needed, integrated into students' coding environment. Empirical deployment in a data science Python course with 82 students shows broad AI hint adoption, with 22% unhelpful hints and 11% escalations; instructor feedback quality is not uniformly high, particularly when AI hints are incorrect. The work contributes a practical, publicly releasable tool architecture and actionable insights into when and why students escalate, guiding future integration of AI and human expertise in education.

Abstract

Timely and high-quality feedback is essential for effective learning in programming courses; yet, providing such support at scale remains a challenge. While AI-based systems offer scalable and immediate help, their responses can occasionally be inaccurate or insufficient. Human instructors, in contrast, may bring more valuable expertise but are limited in time and availability. To address these limitations, we present a hybrid help framework that integrates AI-generated hints with an escalation mechanism, allowing students to request feedback from instructors when AI support falls short. This design leverages the strengths of AI for scale and responsiveness while reserving instructor effort for moments of greatest need. We deployed this tool in a data science programming course with 82 students. We observe that out of the total 673 AI-generated hints, students rated 146 (22%) as unhelpful. Among those, only 16 (11%) of the cases were escalated to the instructors. A qualitative investigation of instructor responses showed that those feedback instances were incorrect or insufficient roughly half of the time. This finding suggests that when AI support fails, even instructors with expertise may need to pay greater attention to avoid making mistakes. We will publicly release the tool for broader adoption and enable further studies in other classrooms. Our work contributes a practical approach to scaling high-quality support and informs future efforts to effectively integrate AI and humans in education.

Closing the Loop: An Instructor-in-the-Loop AI Assistance System for Supporting Student Help-Seeking in Programming Education

TL;DR

The paper tackles the challenge of providing timely, high-quality feedback in programming education at scale. It introduces a hybrid instructor-in-the-loop system that delivers AI-generated hints and escalates to human instructors when needed, integrated into students' coding environment. Empirical deployment in a data science Python course with 82 students shows broad AI hint adoption, with 22% unhelpful hints and 11% escalations; instructor feedback quality is not uniformly high, particularly when AI hints are incorrect. The work contributes a practical, publicly releasable tool architecture and actionable insights into when and why students escalate, guiding future integration of AI and human expertise in education.

Abstract

Timely and high-quality feedback is essential for effective learning in programming courses; yet, providing such support at scale remains a challenge. While AI-based systems offer scalable and immediate help, their responses can occasionally be inaccurate or insufficient. Human instructors, in contrast, may bring more valuable expertise but are limited in time and availability. To address these limitations, we present a hybrid help framework that integrates AI-generated hints with an escalation mechanism, allowing students to request feedback from instructors when AI support falls short. This design leverages the strengths of AI for scale and responsiveness while reserving instructor effort for moments of greatest need. We deployed this tool in a data science programming course with 82 students. We observe that out of the total 673 AI-generated hints, students rated 146 (22%) as unhelpful. Among those, only 16 (11%) of the cases were escalated to the instructors. A qualitative investigation of instructor responses showed that those feedback instances were incorrect or insufficient roughly half of the time. This finding suggests that when AI support fails, even instructors with expertise may need to pay greater attention to avoid making mistakes. We will publicly release the tool for broader adoption and enable further studies in other classrooms. Our work contributes a practical approach to scaling high-quality support and informs future efforts to effectively integrate AI and humans in education.
Paper Structure (21 sections, 9 figures)

This paper contains 21 sections, 9 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tool Design
  4. AI Hint Generation Backend
  5. Student Interaction Interface
  6. Instructor Feedback Interface
  7. Classroom Deployment
  8. Results and Discussion
  9. RQ1: Frequency of Escalation Requests
  10. Analysis Setup.
  11. Analysis Setup.
  12. Findings and discussions.
  13. RQ2: Students' Waiting Time for Instructors
  14. Analysis setup.
  15. Findings and discussions.
  16. ...and 6 more sections

Figures (9)

  • Figure 1: Illustrative example. From top to bottom: Task description; Student's code and comments for requesting a debugging hint; AI-generated hint, which student then rated unhelpful; Student's notes for instructors when escalating for their feedback; and Instructor's feedback, which is more friendly, better targeted to student's issue, and contains reference to online resource.
  • Figure 2:
  • Figure 3: Student interface for requesting AI hints. They can click the ? button to view a description of hint types, the three hint buttons for requesting AI hints, and the collapsible widgets to view received AI hints and instructor feedback.
  • Figure 4: Instructor interface for giving feedback as a single-page website. On the left is the student's notebook. On the right is other contextual information and a place for the instructor to input their feedback.
  • Figure 5: Help requests by type
  • ...and 4 more figures