The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
James Prather, Brent Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S. Randrianasolo, Brett Becker, Bailey Kimmel, Jared Wright, Ben Briggs
TL;DR
This study examines how Generative AI (GenAI) tools affect novice programmers’ metacognition by replicating a prior laboratory study with added eye-tracking. The findings show that GenAI accelerates learning for some students but can worsen metacognitive difficulties for others, introducing new issues such as Interruption, Mislead, and Progression and fostering an illusion of competence. The authors argue that without scaffolding—explicitly teaching problem-solving processes and metacognitive strategies—GenAI may widen the gap between well-prepared and under-prepared novices. They propose interventions like code replays, reflective practices, and scaffolded GenAI experiences to harness benefits while mitigating harms. Overall, the work highlights the need for thoughtful integration of GenAI into computing education and paves the way for targeted pedagogical scaffolds and further replication studies.
Abstract
Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Previous research has shown that novices can encounter multiple metacognitive difficulties while programming. Novices are typically unaware of how these difficulties are hindering their progress. Meanwhile, many novices are now programming with generative AI (GenAI), which can provide complete solutions to most introductory programming problems, code suggestions, hints for next steps when stuck, and explain cryptic error messages. Its impact on novice metacognition has only started to be explored. Here we replicate a previous study that examined novice programming problem solving behavior and extend it by incorporating GenAI tools. Through 21 lab sessions consisting of participant observation, interview, and eye tracking, we explore how novices are coding with GenAI tools. Although 20 of 21 students completed the assigned programming problem, our findings show an unfortunate divide in the use of GenAI tools between students who accelerated and students who struggled. Students who accelerated were able to use GenAI to create code they already intended to make and were able to ignore unhelpful or incorrect inline code suggestions. But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties. Furthermore, struggling students often expressed cognitive dissonance about their problem solving ability, thought they performed better than they did, and finished with an illusion of competence. Based on our observations from both groups, we propose ways to scaffold the novice GenAI experience and make suggestions for future work.