Desirable Characteristics for AI Teaching Assistants in Programming Education

TL;DR

The paper investigates desirable characteristics for AI teaching assistants in introductory programming education by deploying an LLM-powered digital tutor (CodeHelp) in a large university course and surveying 813 students. It finds that students prize instant, engaging support that preserves their agency through scaffolding rather than direct code provision, especially near deadlines. Three core themes emerge—scaffolding, appropriateness, and user experience—guiding design recommendations to tailor guidance to learners' expertise, maintain ownership of problem-solving, and ensure accurate, accessible feedback. The study suggests AI TAs can augment human TAs to provide scalable, around-the-clock support, while highlighting considerations for learning outcomes, procrastination, and ethics in future work.

Abstract

Providing timely and personalized feedback to large numbers of students is a long-standing challenge in programming courses. Relying on human teaching assistants (TAs) has been extensively studied, revealing a number of potential shortcomings. These include inequitable access for students with low confidence when needing support, as well as situations where TAs provide direct solutions without helping students to develop their own problem-solving skills. With the advent of powerful large language models (LLMs), digital teaching assistants configured for programming contexts have emerged as an appealing and scalable way to provide instant, equitable, round-the-clock support. Although digital TAs can provide a variety of help for programming tasks, from high-level problem solving advice to direct solution generation, the effectiveness of such tools depends on their ability to promote meaningful learning experiences. If students find the guardrails implemented in digital TAs too constraining, or if other expectations are not met, they may seek assistance in ways that do not help them learn. Thus, it is essential to identify the features that students believe make digital teaching assistants valuable. We deployed an LLM-powered digital assistant in an introductory programming course and collected student feedback ($n=813$) on the characteristics of the tool they perceived to be most important. Our results highlight that students value such tools for their ability to provide instant, engaging support, particularly during peak times such as before assessment deadlines. They also expressed a strong preference for features that enable them to retain autonomy in their learning journey, such as scaffolding that helps to guide them through problem-solving steps rather than simply being shown direct solutions.

Figures (8)

• Figure 1: Task 1 -- writing code for the IsPrime() function
• Figure 2: Example response from CodeHelp to Task 1.
• Figure 3: Task 2 -- debugging the IsRepeated() function
• Figure 4: Task 3 -- debugging the SurroundingSum() function
• Figure 5: Example response from CodeHelp to Task 2.