Student-AI Interaction: A Case Study of CS1 students
Matin Amoozadeh, Daye Nam, Daniel Prol, Ali Alfageeh, James Prather, Michael Hilton, Sruti Srinivasa Ragavan, Mohammad Amin Alipour
TL;DR
This case study investigates CS1 students' interactions with Generative AI (via an in-editor GPT-3.5 plugin) while solving Python tasks to understand how, why, and when students use AI and how it affects their self-efficacy. It employs a mixed-method design with 15 participants, in-task logging, video/audio data, and pre/post self-efficacy surveys to analyze frequency, interaction patterns, and strategies. Findings show AI-assisted solutions achieved 65% correctness across AI-assisted submissions, with usage occurring early, in the middle, or after solving, and a mix of linear and iterative problem-solving sequences; prompts varied from full-task offloading to targeted conceptual planning and debugging. Self-efficacy outcomes were mixed, with some students improving and others showing declines, underscoring the nuanced impact of GenAI on learning and the need for scaffolded, self-regulated AI usage. The study informs computing education by highlighting design principles to balance AI support with independent problem-solving and proposes pedagogy and tool design to foster productive collaboration with AI in introductory programming.
Abstract
The new capabilities of generative artificial intelligence tools Generative AI, such as ChatGPT, allow users to interact with the system in intuitive ways, such as simple conversations, and receive (mostly) good-quality answers. These systems can support students' learning objectives by providing accessible explanations and examples even with vague queries. At the same time, they can encourage undesired help-seeking behaviors by providing solutions to the students' homework. Therefore, it is important to better understand how students approach such tools and the potential issues such approaches might present for the learners. In this paper, we present a case study for understanding student-AI collaboration to solve programming tasks in the CS1 introductory programming course. To this end, we recruited a gender-balanced majority non-white set of 15 CS1 students at a large public university in the US. We observed them solving programming tasks. We used a mixed-method approach to study their interactions as they tackled Python programming tasks, focusing on when and why they used ChatGPT for problem-solving. We analyze and classify the questions submitted by the 15 participants to ChatGPT. Additionally, we analyzed user interaction patterns, their reactions to ChatGPT's responses, and the potential impacts of Generative AI on their perception of self-efficacy. Our results suggest that in about a third of the cases, the student attempted to complete the task by submitting the full description of the tasks to ChatGPT without making any effort on their own. We also observed that few students verified their solutions. We discuss the results and their potential implications.