How Students Use AI Feedback Matters: Experimental Evidence on Physics Achievement and Autonomy
Xusheng Dai, Zhaochun Wen, Jianxiao Jiang, Huiqin Liu, Yu Zhang
TL;DR
This study investigates how patterns of using GAI-powered feedback influence high school physics achievement and learner autonomy. Across two five-week randomized trials (n=387 total), the researchers compare compulsory AI-assisted recommendations with autonomous on-demand help, revealing heterogeneous effects across achievement groups. Low-achieving students gain substantially from AI hints under compulsory use, while high-achieving students benefit more from self-directed, on-demand feedback; autonomy trends show accompanying declines in certain subgroups, highlighting risks of AI dependence. The findings emphasize that the value of AI feedback depends not only on content quality but crucially on how usage patterns align with learner profiles, with important implications for the design of AI-supported educational tools and policies to balance performance with autonomous learning.
Abstract
Despite the precision and adaptiveness of generative AI (GAI)-powered feedback provided to students, existing practice and literature might ignore how usage patterns impact student learning. This study examines the heterogeneous effects of GAI-powered personalized feedback on high school students' physics achievement and autonomy through two randomized controlled trials, with a major focus on usage patterns. Each experiment lasted for five weeks, involving a total of 387 students. Experiment 1 (n = 121) assessed compulsory usage of the personalized recommendation system, revealing that low-achieving students significantly improved academic performance (d = 0.673, p < 0.05) when receiving AI-generated heuristic solution hints, whereas medium-achieving students' performance declined (d = -0.539, p < 0.05) with conventional answers provided by workbook. Notably, high-achieving students experienced a significant decline in self-regulated learning (d = -0.477, p < 0.05) without any significant gains in achievement. Experiment 2 (n = 266) investigated the usage pattern of autonomous on-demand help, demonstrating that fully learner-controlled AI feedback significantly enhanced academic performance for high-achieving students (d = 0.378, p < 0.05) without negatively impacting their autonomy. However, autonomy notably declined among lower achievers exposed to on-demand AI interventions (d = -0.383, p < 0.05), particularly in the technical-psychological dimension (d = -0.549, p < 0.05), which has a large overlap with self-regulation. These findings underscore the importance of usage patterns when applying GAI-powered personalized feedback to students.