Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revise
Keno Harada, Lui Yoshida, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo
TL;DR
The paper addresses the misalignment between LLM-based automated essay scoring and human judgments by introducing an iterative, reflect-and-revise rubric refinement approach. By having models reflect on scoring rationales and observed discrepancies, the method gradually updates rubrics to improve agreement, evaluated on TOEFL11 and ASAP with multiple models. Key findings show significant improvements in Quadratic Weighted Kappa, with up to 0.19 on TOEFL11 and 0.47 on ASAP, and crucially, even a very simple initial rubric can outperform detailed human rubrics after refinement. This work highlights a scalable, model-driven path to calibrate AES rubrics, reducing dependence on expertly crafted rubrics while enhancing alignment with human evaluation.
Abstract
The performance of Large Language Models (LLMs) is highly sensitive to the prompts they are given. Drawing inspiration from the field of prompt optimization, this study investigates the potential for enhancing Automated Essay Scoring (AES) by refining the scoring rubrics used by LLMs. Specifically, our approach prompts models to iteratively refine rubrics by reflecting on models' own scoring rationales and observed discrepancies with human scores on sample essays. Experiments on the TOEFL11 and ASAP datasets using GPT-4.1, Gemini-2.5-Pro, and Qwen-3-Next-80B-A3B-Instruct show Quadratic Weighted Kappa (QWK) improvements of up to 0.19 and 0.47, respectively. Notably, even with a simple initial rubric, our approach achieves comparable or better QWK than using detailed human-authored rubrics. Our findings highlight the importance of iterative rubric refinement in LLM-based AES to enhance alignment with human evaluations.