Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

61A Bot Report: AI Assistants in CS1 Save Students Homework Time and Reduce Demands on Staff. (Now What?)

J. D. Zamfirescu-Pereira, Laryn Qi, Björn Hartmann, John DeNero, Narges Norouzi

TL;DR

This study documents the deployment of a GPT-4–based interactive homework assistant, 61A Bot, embedded within the autograder for UC Berkeley's CS 61A. Across two semesters and over 2,000 students, the Bot provides one-shot, contextual feedback during code testing, with a design that avoids giving direct solutions. The authors report a dramatic reduction in forum questions and substantial reductions in homework completion times—up to 25–50% faster for mid-range performers—while labs show little to no speedup, and projects show modest improvements. While learning outcomes are not conclusively improved, the work highlights both the potential benefits (time savings, staff relief) and costs (risks to error-trace engagement, governance considerations) of integrating AI assistance in CS1, and outlines directions for future research on learning impacts and scalable, responsible deployment.

Abstract

LLM-based chatbots enable students to get immediate, interactive help on homework assignments, but even a thoughtfully-designed bot may not serve all pedagogical goals. We report here on the development and deployment of a GPT-4-based interactive homework assistant ("61A Bot") for students in a large CS1 course; over 2000 students made over 100,000 requests of our Bot across two semesters. Our assistant offers one-shot, contextual feedback within the command-line "autograder" students use to test their code. Our Bot wraps student code in a custom prompt that supports our pedagogical goals and avoids providing solutions directly. Analyzing student feedback, questions, and autograder data, we find reductions in homework-related question rates in our course forum, as well as reductions in homework completion time when our Bot is available. For students in the 50th-80th percentile, reductions can exceed 30 minutes per assignment, up to 50% less time than students at the same percentile rank in prior semesters. Finally, we discuss these observations, potential impacts on student learning, and other potential costs and benefits of AI assistance in CS1.

61A Bot Report: AI Assistants in CS1 Save Students Homework Time and Reduce Demands on Staff. (Now What?)

TL;DR

This study documents the deployment of a GPT-4–based interactive homework assistant, 61A Bot, embedded within the autograder for UC Berkeley's CS 61A. Across two semesters and over 2,000 students, the Bot provides one-shot, contextual feedback during code testing, with a design that avoids giving direct solutions. The authors report a dramatic reduction in forum questions and substantial reductions in homework completion times—up to 25–50% faster for mid-range performers—while labs show little to no speedup, and projects show modest improvements. While learning outcomes are not conclusively improved, the work highlights both the potential benefits (time savings, staff relief) and costs (risks to error-trace engagement, governance considerations) of integrating AI assistance in CS1, and outlines directions for future research on learning impacts and scalable, responsible deployment.

Abstract

LLM-based chatbots enable students to get immediate, interactive help on homework assignments, but even a thoughtfully-designed bot may not serve all pedagogical goals. We report here on the development and deployment of a GPT-4-based interactive homework assistant ("61A Bot") for students in a large CS1 course; over 2000 students made over 100,000 requests of our Bot across two semesters. Our assistant offers one-shot, contextual feedback within the command-line "autograder" students use to test their code. Our Bot wraps student code in a custom prompt that supports our pedagogical goals and avoids providing solutions directly. Analyzing student feedback, questions, and autograder data, we find reductions in homework-related question rates in our course forum, as well as reductions in homework completion time when our Bot is available. For students in the 50th-80th percentile, reductions can exceed 30 minutes per assignment, up to 50% less time than students at the same percentile rank in prior semesters. Finally, we discuss these observations, potential impacts on student learning, and other potential costs and benefits of AI assistance in CS1.
Paper Structure (15 sections, 3 figures, 1 table)

This paper contains 15 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Design & Deployment
  4. Course Details & Deployment
  5. Using 61A Bot
  6. Deployment Timeline
  7. Outcomes
  8. Usage & Reliance on Staff
  9. Student Reception
  10. Effects on Homework
  11. Data & Analytical Method
  12. Results
  13. Discussion & Future Work
  14. Limitations & Threats to Validity
  15. Conclusions

Figures (3)

  • Figure 1: Screenshot of the "autograder" command-line interface. Students run the autograder to test their code against a test suite (1); if any test cases fail, they are asked if they would like to receive Bot feedback, and then asked to assess the hint (2).
  • Figure 2: 61A Bot Survey Results. Students were asked to report on their usage and perceptions of Bot helpfulness and reliability, as well as their overall satisfaction on a scale of 1-5, from least to most.
  • Figure 3: CDFs of assignment completion times. Circles identify the 20, 40, 60, and 80th percentiles; differences can be read by matching circle markers horizontally. E.g., for HW2, the mean 60th percentile time across SP21-SP23 is 34 minutes; in SP24 this time is 21 minutes, a 38% reduction. $^*$ denotes differences significant at the $p < 0.001$ level.