Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FeedEval: Pedagogically Aligned Evaluation of LLM-Generated Essay Feedback

Seongyeub Chu, Jongwoo Kim, Munyong Yi

TL;DR

FeedEval introduces a pedagogy-grounded framework to evaluate LLM-generated essay feedback along specificity, helpfulness, and validity, addressing noise in feedback used for training essay assessment models. It builds three specialized evaluators trained on curated datasets (SpecEval for specificity, a helpfulness dataset from RECIPE4U/FEAT/ASAP++, and a Prometheus-based NLI validity evaluator) to filter high-quality feedback from multiple candidates. On ASAP++, FeedEval demonstrations show strong alignment with human experts and improved essay-scoring performance when trained on Filtered high-quality feedback, as well as more effective revisions with such feedback from small LLMs. The work also generalizes to ASAP-SAS and provides dataset resources, highlighting the potential of pedagogy-aware feedback evaluation to enhance LLM-based educational tools while noting limitations like generation order effects and language scope.

Abstract

Going beyond the prediction of numerical scores, recent research in automated essay scoring has increasingly emphasized the generation of high-quality feedback that provides justification and actionable guidance. To mitigate the high cost of expert annotation, prior work has commonly relied on LLM-generated feedback to train essay assessment models. However, such feedback is often incorporated without explicit quality validation, resulting in the propagation of noise in downstream applications. To address this limitation, we propose FeedEval, an LLM-based framework for evaluating LLM-generated essay feedback along three pedagogically grounded dimensions: specificity, helpfulness, and validity. FeedEval employs dimension-specialized LLM evaluators trained on datasets curated in this study to assess multiple feedback candidates and select high-quality feedback for downstream use. Experiments on the ASAP++ benchmark show that FeedEval closely aligns with human expert judgments and that essay scoring models trained with FeedEval-filtered high-quality feedback achieve superior scoring performance. Furthermore, revision experiments using small LLMs show that the high-quality feedback identified by FeedEval leads to more effective essay revisions. We will release our code and curated datasets upon accepted.

FeedEval: Pedagogically Aligned Evaluation of LLM-Generated Essay Feedback

TL;DR

FeedEval introduces a pedagogy-grounded framework to evaluate LLM-generated essay feedback along specificity, helpfulness, and validity, addressing noise in feedback used for training essay assessment models. It builds three specialized evaluators trained on curated datasets (SpecEval for specificity, a helpfulness dataset from RECIPE4U/FEAT/ASAP++, and a Prometheus-based NLI validity evaluator) to filter high-quality feedback from multiple candidates. On ASAP++, FeedEval demonstrations show strong alignment with human experts and improved essay-scoring performance when trained on Filtered high-quality feedback, as well as more effective revisions with such feedback from small LLMs. The work also generalizes to ASAP-SAS and provides dataset resources, highlighting the potential of pedagogy-aware feedback evaluation to enhance LLM-based educational tools while noting limitations like generation order effects and language scope.

Abstract

Going beyond the prediction of numerical scores, recent research in automated essay scoring has increasingly emphasized the generation of high-quality feedback that provides justification and actionable guidance. To mitigate the high cost of expert annotation, prior work has commonly relied on LLM-generated feedback to train essay assessment models. However, such feedback is often incorporated without explicit quality validation, resulting in the propagation of noise in downstream applications. To address this limitation, we propose FeedEval, an LLM-based framework for evaluating LLM-generated essay feedback along three pedagogically grounded dimensions: specificity, helpfulness, and validity. FeedEval employs dimension-specialized LLM evaluators trained on datasets curated in this study to assess multiple feedback candidates and select high-quality feedback for downstream use. Experiments on the ASAP++ benchmark show that FeedEval closely aligns with human expert judgments and that essay scoring models trained with FeedEval-filtered high-quality feedback achieve superior scoring performance. Furthermore, revision experiments using small LLMs show that the high-quality feedback identified by FeedEval leads to more effective essay revisions. We will release our code and curated datasets upon accepted.
Paper Structure (62 sections, 10 figures, 19 tables, 1 algorithm)

This paper contains 62 sections, 10 figures, 19 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-based Essay Assessment
  4. Evaluation of LLM-generated Feedback
  5. FeedEval Framework
  6. Evaluation Dimensions
  7. Evaluation Pipeline
  8. Specificity Evaluator
  9. Helpfulness Evaluator
  10. Validity Evaluator
  11. Experimental Setup
  12. Datasets
  13. Human Expert Alignment of FeedEval
  14. Feedback Quality Evaluation
  15. Impact of Feedback Quality on Essay Scoring
  16. ...and 47 more sections

Figures (10)

  • Figure 1: FeedEval evaluates the quality of multiple feedback candidates for the same essay by assessing how well they reference the essay, align with the rubric, and provide actionable revision suggestions.
  • Figure 2: Overview of the proposed FeedEval framework. FeedEval consists of three evaluators, each trained on a dataset corresponding to a specific evaluation dimension. Given multiple feedback candidates generated by an LLM, FeedEval evaluates their quality along three dimensions and selects the highest-quality feedback.
  • Figure 3: Average essay scoring performance across traits on ASAP++ for Qwen3-8B trained with high-quality feedback labels filtered by FeedEval using one, two, or all three dimensions.
  • Figure 4: Average essay score improvement across traits on ASAP++ after revisions guided by feedback of high- and low-quality identified by FeedEval and GPT-5.1.
  • Figure 5: Prompt template for feedback generation using both human-annotated scores and score descriptions of the rubric.
  • ...and 5 more figures