Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs

Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Z. Henley, Paul Denny, Michelle Craig, Tovi Grossman

TL;DR

The paper tackles scalable, equitable programming education by introducing CodeAid, an LLM-based assistant that provides conceptual guidance, pseudo-code scaffolding, and targeted code annotations without exposing direct solutions. Through a 12-week deployment in a 700-student CS course, the authors collect multi-source data (8,000+ interactions, surveys, interviews) and perform thematic analyses to assess usage, answer quality, and user trust. Key contributions include four design considerations for educational AI tooling, a robust multi-faceted evaluation, and iterative system updates such as pseudo-code integration and line-level explanations. The work demonstrates that pedagogically guarded AI tools can augment learning, reduce dependence on direct solutions, and offer actionable insights for integrating AI assistants into curricula while preserving instructional integrity and equity.

Abstract

Timely, personalized feedback is essential for students learning programming. LLM-powered tools like ChatGPT offer instant support, but reveal direct answers with code, which may hinder deep conceptual engagement. We developed CodeAid, an LLM-powered programming assistant delivering helpful, technically correct responses, without revealing code solutions. CodeAid answers conceptual questions, generates pseudo-code with line-by-line explanations, and annotates student's incorrect code with fix suggestions. We deployed CodeAid in a programming class of 700 students for a 12-week semester. A thematic analysis of 8,000 usages of CodeAid was performed, further enriched by weekly surveys, and 22 student interviews. We then interviewed eight programming educators to gain further insights. Our findings reveal four design considerations for future educational AI assistants: D1) exploiting AI's unique benefits; D2) simplifying query formulation while promoting cognitive engagement; D3) avoiding direct responses while encouraging motivated learning; and D4) maintaining transparency and control for students to asses and steer AI responses.

CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs

TL;DR

The paper tackles scalable, equitable programming education by introducing CodeAid, an LLM-based assistant that provides conceptual guidance, pseudo-code scaffolding, and targeted code annotations without exposing direct solutions. Through a 12-week deployment in a 700-student CS course, the authors collect multi-source data (8,000+ interactions, surveys, interviews) and perform thematic analyses to assess usage, answer quality, and user trust. Key contributions include four design considerations for educational AI tooling, a robust multi-faceted evaluation, and iterative system updates such as pseudo-code integration and line-level explanations. The work demonstrates that pedagogically guarded AI tools can augment learning, reduce dependence on direct solutions, and offer actionable insights for integrating AI assistants into curricula while preserving instructional integrity and equity.

Abstract

Timely, personalized feedback is essential for students learning programming. LLM-powered tools like ChatGPT offer instant support, but reveal direct answers with code, which may hinder deep conceptual engagement. We developed CodeAid, an LLM-powered programming assistant delivering helpful, technically correct responses, without revealing code solutions. CodeAid answers conceptual questions, generates pseudo-code with line-by-line explanations, and annotates student's incorrect code with fix suggestions. We deployed CodeAid in a programming class of 700 students for a 12-week semester. A thematic analysis of 8,000 usages of CodeAid was performed, further enriched by weekly surveys, and 22 student interviews. We then interviewed eight programming educators to gain further insights. Our findings reveal four design considerations for future educational AI assistants: D1) exploiting AI's unique benefits; D2) simplifying query formulation while promoting cognitive engagement; D3) avoiding direct responses while encouraging motivated learning; and D4) maintaining transparency and control for students to asses and steer AI responses.
Paper Structure (71 sections, 7 figures, 6 tables)

This paper contains 71 sections, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs in Computer Science Education
  4. Question Answering
  5. Explaining Code
  6. Writing and Debugging Code
  7. Initial System Design and Architecture
  8. Primary CodeAid Assistance Features
  9. General Question
  10. Inline Code Exploration
  11. Question from Code
  12. Help Fix Code
  13. Explain Code
  14. Help Write Code
  15. System Architecture
  16. ...and 56 more sections

Figures (7)

  • Figure 1: The primary input interface of CodeAid. Users select a feature from the bottom right; this choice activates the relevant input fields (code or question). After inputting their query, users press 'ask' and wait for the LLM to respond.
  • Figure 2: The initial interface for the responses produced by CodeAid's five primary functions, along with the Inline Code Exploration feature.
  • Figure 3: The structure of LLM prompts used in the initial version of General Question, Help Fix Code, and Explain Code.
  • Figure 4: The redesigned interface for the responses produced by CodeAid after the midterm update: (a) redesigned Help Fix Code, (b) the new interactive pseudo-code with line-by-line explanations, (c) suggested follow-ups, and (d) displaying relevant function definitions.
  • Figure 5: Revised prompt design and system architecture illustrating the data flow of the General Question feature, highlighting the process of generating pseudo-code and relevant function documentation.
  • ...and 2 more figures