Changes in Coding Behavior and Performance Since the Introduction of LLMs

TL;DR

The paper investigates how widespread LLM usage since late 2022 reshapes student coding behavior and learning in a graduate CS course by longitudinally analyzing 2,066 PageRank submissions across 10 semesters. Using fixed-task auto-grading and metrics on coding activity (Total and Average Edit Distance, Number of Submissions) and performance (Task Score, IP Score, TP Score), it finds significantly longer and more edited final submissions post-LLMs while maintaining high Task Scores and a slight dip in IP Scores, with TP Scores rising. A negative association between more extensive edits and IP outcomes is observed ($r = -0.16$, $p < 0.001$), suggesting over-reliance on AI tools may erode individual learning despite stable or improved team performance. The work advocates rethinking CS education around human–AI collaboration (the centaur model) and signals implications for how we assess coding expertise and productivity in both academia and industry.

Abstract

The widespread availability of large language models (LLMs) has changed how students engage with coding and problem-solving. While these tools may increase student productivity, they also make it more difficult for instructors to assess students' learning and effort. In this quasi-longitudinal study, we analyze five years of student source code submissions in a graduate-level cloud computing course, focusing on an assignment that remained unchanged and examining students' behavior during the period spanning five semesters before the release of ChatGPT and five semesters after. Student coding behavior has changed significantly since Fall 2022. The length of their final submissions increased. Between consecutive submissions, average edit distances increased while average score improvement decreased, suggesting that both student productivity and learning have decreased after ChatGPT's release. Additionally, there are statistically significant correlations between these behavioral changes and their overall performance. Although we cannot definitively attribute them to LLM misuse, they are consistent with our hypothesis that some students are over-reliant on LLMs, which is negatively affecting their learning outcomes. Our findings raise an alarm around the first generation of graduates in the age of LLMs, calling upon both educators and employers to reflect on their evaluation methods for genuine expertise and productivity.

Figures (13)

• Figure 1: Lines of code in final, perfect-score submissions increased significantly since Fall 2022. Note: the y-axis is not zero-indexed.
• Figure 2: Enrollments by semester, split by participation in the PageRank task.
• Figure 3: Enrollments by semester, split by experience Level.
• Figure 4: Number of submissions per student ticked up slightly since s23.
• Figure 5: Total edit distance increased since s23