Amplifiers or Equalizers? A Longitudinal Study of LLM Evolution in Software Engineering Project-Based Learning
Hana Kataoka, Jialong Li, Yutaka Matsuno
TL;DR
The paper investigates how rapid LLM evolution reshapes project-based software engineering education by comparing a 2024 cohort using free LLMs with a 2025 cohort using paid, more capable LLMs. Using COSMIC Function Points, CFRP, and test-case pass rates, it shows that stronger LLMs can elevate project complexity even for weaker students (equalizer) while widening absolute performance gaps (amplifier). The findings highlight dual pedagogical implications: enabling richer SE practices within limited time, but exacerbating inequities without targeted interventions. The work suggests redesigning LLM-era IPBL and outlines future directions for artifact analysis, interaction-process mining, and expanded data collection to inform equitable teaching strategies.
Abstract
As LLMs reshape software development, integrating LLM-augmented practices into SE education has become imperative. While existing studies explore LLMs' educational use in introductory programming or isolated SE tasks, their impact in more open-ended Project-Based Learning (PBL) remains unexplored. This paper introduces a two-year longitudinal study comparing a 2024 (using early free LLMs, $n$=48) and 2025 (using the latest paid LLMs, $n$=46) cohort. Our findings suggest the latest powerful LLMs' dual role: they act as "equalizers," boosting average performance even for programming-weak students, providing opportunities for more authentic SE practices; yet also as "amplifiers," dramatically widening absolute performance gaps, creating new pedagogical challenges for addressing educational inequities.