AI-Mediated Feedback Improves Student Revisions: A Randomized Trial with FeedbackWriter in a Large Undergraduate Course

TL;DR

The paper tests FeedbackWriter, an AI-assisted feedback system designed to complement human TAs in a large undergraduate economics course. Through a randomized trial (N=354) across two writing assignments, AI-mediated feedback yields significantly higher-quality revisions (effect size $d=0.50$) and greater rubric coverage, though post-test learning gains do not differ from the human-only baseline. TAs actively exploit AI suggestions, often editing and augmenting AI outputs, and report enhanced efficiency, coherence, and targeted guidance, while maintaining control to prevent overreliance on AI. The findings support a human-AI collaboration approach where well-structured rubrics, integrated interfaces, and TA oversight enable scalable, high-quality feedback for knowledge-intensive writing. The work provides design principles for rubric refinement, user-centered interfaces, and cautions about the limitations and ethical considerations of AI-assisted feedback in education.

Abstract

Despite growing interest in using LLMs to generate feedback on students' writing, little is known about how students respond to AI-mediated versus human-provided feedback. We address this gap through a randomized controlled trial in a large introductory economics course (N=354), where we introduce and deploy FeedbackWriter - a system that generates AI suggestions to teaching assistants (TAs) while they provide feedback on students' knowledge-intensive essays. TAs have the full capacity to adopt, edit, or dismiss the suggestions. Students were randomly assigned to receive either handwritten feedback from TAs (baseline) or AI-mediated feedback where TAs received suggestions from FeedbackWriter. Students revise their drafts based on the feedback, which is further graded. In total, 1,366 essays were graded using the system. We found that students receiving AI-mediated feedback produced significantly higher-quality revisions, with gains increasing as TAs adopted more AI suggestions. TAs found the AI suggestions useful for spotting gaps and clarifying rubrics.

Figures (12)

• Figure 1: The design of FeedbackWriter follows TAs' natural workflow of feedback provision. AI suggestions are provided for each rubric and are surfaced through corresponding UI elements, including 1) identifying relevant sentences (sentence highlighting); 2) judging whether the rubric is adequately and accurately addressed (rubric judgments); and 3) constructing effective feedback (AI feedback and historic feedback).
• Figure 2: FeedbackWriter interface contains an essay panel (left) and a feedback panel (right). The essay panel displays the student essay, where the relevant sentences associated with each feedback are highlighted (A). The feedback panel displays AI-generated comments organized by rubrics. Each rubric item has its own feedback box, including the rubric, rubric judgment with color cues (B) and two feedback suggestions (C). Users can also manually add additional in-text comments (D).
• Figure 3: In FeedbackWriter, users can use and adjust all AI suggestions conveniently. Users can use (A) the "reposition" button to modify the highlight, click (B) to flip the judgment, and click (C) the "regenerate" button to get another suggestion. The feedback boxes are empty by default, and users can click (D) to adopt either feedback suggestion.
• Figure 4: FeedbackWriter further supports the TAs on grading the final drafts in two ways. (A) a collapsible "diff" panel that visualizes students' additions and deletions between the first and final drafts; (B) feedback from the first draft on the same rubric item is shown in the feedback box.
• Figure 5: Timeline for the study. For each assignment in ECON101, students submitted a first draft, received and incorporated TAs' feedback to form a final draft. The conditions were counter-balanced for the assignments.