Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring LLM-Generated Feedback for Economics Essays: How Teaching Assistants Evaluate and Envision Its Use

Xinyi Lu, Aditya Mahesh, Zejia Shen, Mitchell Dudley, Larissa Sano, Xu Wang

TL;DR

The paper investigates whether LLM-generated feedback can serve as actionable suggestions to speed and improve human instructors' feedback on knowledge-intensive economics essays. It introduces a modular feedback pipeline that identifies relevant essay sentences per rubric, assesses rubric satisfaction, and generates rubric-aligned feedback, visualized in a Word plugin with highlights. Through think-aloud studies with five TAs across four assignments, the study finds AI feedback often mirrors effective feedback features and can be more rubric-aligned and personalized than historic feedback, though it risks rigidity and fragmentation; TAs view AI suggestions as a way to accelerate grading and improve consistency when used collaboratively with human judgment. Design implications emphasize detailed rubrics, transparent highlighting of AI reasoning, and intermediate AI outputs to support trustworthy human-AI collaboration, with broader potential to scale feedback quality in knowledge-intensive courses. Limitations include hallucinations and edge cases not captured by rubrics; future work should assess student learning outcomes and extend the approach to other disciplines and course contexts.

Abstract

This project examines the prospect of using AI-generated feedback as suggestions to expedite and enhance human instructors' feedback provision. In particular, we focus on understanding the teaching assistants' perspectives on the quality of AI-generated feedback and how they may or may not utilize AI feedback in their own workflows. We situate our work in a foundational college Economics class, which has frequent short essay assignments. We developed an LLM-powered feedback engine that generates feedback on students' essays based on grading rubrics used by the teaching assistants (TAs). To ensure that TAs can meaningfully critique and engage with the AI feedback, we had them complete their regular grading jobs. For a randomly selected set of essays that they had graded, we used our feedback engine to generate feedback and displayed the feedback as in-text comments in a Word document. We then performed think-aloud studies with 5 TAs over 20 1-hour sessions to have them evaluate the AI feedback, contrast the AI feedback with their handwritten feedback, and share how they envision using the AI feedback if they were offered as suggestions. The study highlights the importance of providing detailed rubrics for AI to generate high-quality feedback for knowledge-intensive essays. TAs considered that using AI feedback as suggestions during their grading could expedite grading, enhance consistency, and improve overall feedback quality. We discuss the importance of decomposing the feedback generation task into steps and presenting intermediate results, in order for TAs to use the AI feedback.

Exploring LLM-Generated Feedback for Economics Essays: How Teaching Assistants Evaluate and Envision Its Use

TL;DR

The paper investigates whether LLM-generated feedback can serve as actionable suggestions to speed and improve human instructors' feedback on knowledge-intensive economics essays. It introduces a modular feedback pipeline that identifies relevant essay sentences per rubric, assesses rubric satisfaction, and generates rubric-aligned feedback, visualized in a Word plugin with highlights. Through think-aloud studies with five TAs across four assignments, the study finds AI feedback often mirrors effective feedback features and can be more rubric-aligned and personalized than historic feedback, though it risks rigidity and fragmentation; TAs view AI suggestions as a way to accelerate grading and improve consistency when used collaboratively with human judgment. Design implications emphasize detailed rubrics, transparent highlighting of AI reasoning, and intermediate AI outputs to support trustworthy human-AI collaboration, with broader potential to scale feedback quality in knowledge-intensive courses. Limitations include hallucinations and edge cases not captured by rubrics; future work should assess student learning outcomes and extend the approach to other disciplines and course contexts.

Abstract

This project examines the prospect of using AI-generated feedback as suggestions to expedite and enhance human instructors' feedback provision. In particular, we focus on understanding the teaching assistants' perspectives on the quality of AI-generated feedback and how they may or may not utilize AI feedback in their own workflows. We situate our work in a foundational college Economics class, which has frequent short essay assignments. We developed an LLM-powered feedback engine that generates feedback on students' essays based on grading rubrics used by the teaching assistants (TAs). To ensure that TAs can meaningfully critique and engage with the AI feedback, we had them complete their regular grading jobs. For a randomly selected set of essays that they had graded, we used our feedback engine to generate feedback and displayed the feedback as in-text comments in a Word document. We then performed think-aloud studies with 5 TAs over 20 1-hour sessions to have them evaluate the AI feedback, contrast the AI feedback with their handwritten feedback, and share how they envision using the AI feedback if they were offered as suggestions. The study highlights the importance of providing detailed rubrics for AI to generate high-quality feedback for knowledge-intensive essays. TAs considered that using AI feedback as suggestions during their grading could expedite grading, enhance consistency, and improve overall feedback quality. We discuss the importance of decomposing the feedback generation task into steps and presenting intermediate results, in order for TAs to use the AI feedback.

Paper Structure

This paper contains 25 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Existing work on evaluating AI feedback
  4. Capabilities of LLMs in modular tasks
  5. Method
  6. Tasks
  7. Feedback generation and visualization
  8. Study Design
  9. Procedure
  10. Data analysis methods
  11. Findings
  12. RQ1: How do TAs perceive the differences between AI-generated feedback and their own feedback?
  13. AI feedback exhibits more characteristics of effective feedback, such as providing praise, using guiding questions and explanations.
  14. AI feedback is more personalized to students' responses in comparison to the historic feedback.
  15. AI-generated feedback aligns more closely with the rubrics, which can be a double-edged sword—it ensures consistency but may be misleading when the AI applies the rubric too rigidly.
  16. ...and 10 more sections

Figures (2)

  • Figure 1: The feedback generation is decomposed into steps. With the relevant sentences for each rubric identified in (a), AI makes judgments on whether each rubric is met (b). Then the relevant sentences and judgments are used to generate feedback (c). Additionally, if the rubric is missed, historic feedback is retrieved from a set of feedback designed by instructors. The feedback and relevant information will be shown as in-text comments in a Word document (d).
  • Figure 2: AI-generated feedback is shown as in-text comments on a Word document, added to the most relevant sentences in the student essay. Each comment contains the rubric, the AI judgment, the AI feedback, and the historic feedback.