Same Feedback, Different Source: How AI vs. Human Feedback Shapes Learner Engagement

TL;DR

The paper investigates whether perceived feedback source—AI versus a human TA—shapes learner engagement when the feedback content is identical. Using a controlled-content, 2×2 mixed design across four progressive p5.js modules, the authors show that learners told the feedback comes from a TA spend substantially more time and effort ($d$ up to $1.56$) than those told it is AI, despite identical content. Feedback ratings are predicted by different factors depending on the attributed source: prior trust in AI drives AI-attributed ratings ($r=0.847$, $p=.001$; $r=0.761$, $p=.007$ for technical), while perceived authenticity drives TA-attributed ratings ($r=0.647$, $p=.012$; $r=0.653$, $p=.011$). These findings support social presence theory and imply that attribution framing can modulate engagement, informing the design of hybrid AI–human educational tools. The study introduces a controlled-content methodology to isolate attribution effects, offering a path to optimize learner engagement in mixed-initiative learning environments.

Abstract

When learners receive feedback, what they believe about its source may shape how they engage with it. As AI is used alongside human instructors, understanding these attribution effects is essential for designing effective hybrid AI-human educational systems. We designed a creative coding interface that isolates source attribution while controlling for content: all participants receive identical LLM-generated feedback, but half see it attributed to AI and half to a human teaching assistant (TA). We found two key results. First, perceived feedback source affected engagement: learners in the TA condition spent significantly more time and effort (d = 0.88-1.56) despite receiving identical feedback. Second, perceptions differed: AI-attributed feedback ratings were predicted by prior trust in AI (r = 0.85), while TA-attributed ratings were predicted by perceived genuineness (r = 0.65). These findings suggest that feedback source shapes both engagement and evaluation, with implications for hybrid educational system design.

Figures (4)

• Figure 1: (Left) Web interface for creative coding tutorial. A three-panel design allows simultaneous access to a vertical-scroll tutorial, p5.js code editor, and graphical output canvas. (Right) Checkpoint modal in web interface. Feedback on technical (L) and creative (R) properties are shown simultaneously, attributed to either an AI or TA by research condition. Participants rate each feedback section for usefulness before proceeding to the following module.
• Figure 2: Sample code, graphical output, and feedback from participants in the AI (top) and Human TA (bottom) feedback conditions. From L-R: Code subset, visual output of a single module's exercise, technical feedback and associated rating from the participant, creative feedback and associated rating from the participant.
• Figure 3: Behavioral engagement by condition. Left: Time spent per module (minutes), with significant differences on modules occurring after initial feedback. Center: Total code length in final project (characters). Right: Code execution frequency across the tutorial.
• Figure 4: Predictors of feedback ratings by condition. (Left) In the AI condition, prior trust in AI strongly predicted ratings. (Right) In the TA condition, perceived genuineness of feedback predicted ratings. Each point represents one participant's mean feedback rating.