Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant

TL;DR

This study investigates how students interact with CodeHelp, an LLM-powered programming assistant with guardrails that prevent direct solution outputs, deployed over a 12-week introductory CS course (n=52). Analyzing 2,591 raw queries (2,082 post-deduplication) revealed that debugging and implementation queries dominated, while attempts to deepen conceptual understanding were less common; many queries were low-effort or copied from course materials. A modest positive association was found between overall CodeHelp usage and course performance ($r = 0.38$, $p = 0.0126$), suggesting potential learning benefits without evidence of harm, though causality cannot be established. The findings inform teaching practices and tool design by highlighting the need to guide students toward higher-quality queries and to balance rapid, task-focused help with deeper conceptual understanding.

Abstract

Providing personalized assistance at scale is a long-standing challenge for computing educators, but a new generation of tools powered by large language models (LLMs) offers immense promise. Such tools can, in theory, provide on-demand help in large class settings and be configured with appropriate guardrails to prevent misuse and mitigate common concerns around learner over-reliance. However, the deployment of LLM-powered tools in authentic classroom settings is still rare, and very little is currently known about how students will use them in practice and what type of help they will seek. To address this, we examine students' use of an innovative LLM-powered tool that provides on-demand programming assistance without revealing solutions directly. We deployed the tool for 12 weeks in an introductory computer and data science course ($n = 52$), collecting more than 2,500 queries submitted by students throughout the term. We manually categorized all student queries based on the type of assistance sought, and we automatically analyzed several additional query characteristics. We found that most queries requested immediate help with programming assignments, whereas fewer requests asked for help on related concepts or for deepening conceptual understanding. Furthermore, students often provided minimal information to the tool, suggesting this is an area in which targeted instruction would be beneficial. We also found that students who achieved more success in the course tended to have used the tool more frequently overall. Lessons from this research can be leveraged by programming educators and institutions who plan to augment their teaching with emerging LLM-powered tools.

Figures (8)

• Figure 1: The Help Request form (text areas have been shrunk here to save space). The four separate inputs (language, code, error, and issue) and connected guidance text help students structure their request and encourage good practices when requesting support.
• Figure 2: Prompt used for the sufficiency check.
• Figure 3: Prompt used for the main response.
• Figure 4: Prompt used for code removal.
• Figure 5: An example of a student's input directly asking for a solution and CodeHelp's response.