Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models Are State-of-the-Art Evaluator for Grammatical Error Correction

Masamune Kobayashi, Masato Mita, Mamoru Komachi

TL;DR

This work investigates large language models as evaluators for English grammatical error correction (GEC) by designing prompts that encode evaluation criteria and by performing comprehensive meta-evaluation against traditional metrics. Using the SEEDA dataset, the authors compare system-level and sentence-level correlations between human judgments and LLM-based scores across EBMs/SBMs, revealing that GPT-4 achieves state-of-the-art alignment, with a Kendall's tau around $0.662$. The study highlights that model scale and fluency-focused prompts substantially improve evaluation performance, while smaller models show reduced sensitivity to fluency and tend to cluster scores. The findings support LLM-based evaluation as a robust, scalable alternative to conventional GEC metrics and outline future work on few-shot prompting and document-level assessment.

Abstract

Large Language Models (LLMs) have been reported to outperform existing automatic evaluation metrics in some tasks, such as text summarization and machine translation. However, there has been a lack of research on LLMs as evaluators in grammatical error correction (GEC). In this study, we investigate the performance of LLMs in GEC evaluation by employing prompts designed to incorporate various evaluation criteria inspired by previous research. Our extensive experimental results demonstrate that GPT-4 achieved Kendall's rank correlation of 0.662 with human judgments, surpassing all existing methods. Furthermore, in recent GEC evaluations, we have underscored the significance of the LLMs scale and particularly emphasized the importance of fluency among evaluation criteria.

Large Language Models Are State-of-the-Art Evaluator for Grammatical Error Correction

TL;DR

This work investigates large language models as evaluators for English grammatical error correction (GEC) by designing prompts that encode evaluation criteria and by performing comprehensive meta-evaluation against traditional metrics. Using the SEEDA dataset, the authors compare system-level and sentence-level correlations between human judgments and LLM-based scores across EBMs/SBMs, revealing that GPT-4 achieves state-of-the-art alignment, with a Kendall's tau around . The study highlights that model scale and fluency-focused prompts substantially improve evaluation performance, while smaller models show reduced sensitivity to fluency and tend to cluster scores. The findings support LLM-based evaluation as a robust, scalable alternative to conventional GEC metrics and outline future work on few-shot prompting and document-level assessment.

Abstract

Large Language Models (LLMs) have been reported to outperform existing automatic evaluation metrics in some tasks, such as text summarization and machine translation. However, there has been a lack of research on LLMs as evaluators in grammatical error correction (GEC). In this study, we investigate the performance of LLMs in GEC evaluation by employing prompts designed to incorporate various evaluation criteria inspired by previous research. Our extensive experimental results demonstrate that GPT-4 achieved Kendall's rank correlation of 0.662 with human judgments, surpassing all existing methods. Furthermore, in recent GEC evaluations, we have underscored the significance of the LLMs scale and particularly emphasized the importance of fluency among evaluation criteria.
Paper Structure (18 sections, 4 figures, 11 tables)

This paper contains 18 sections, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Experiment setup
  3. Considered metrics
  4. GEC metrics:
  5. LLMs:
  6. Meta-evaluation methods
  7. System-level meta-evaluation:
  8. Sentence-level meta-evaluation:
  9. Results
  10. System-level analysis
  11. Sentence-level analysis
  12. Further analysis
  13. Related Work
  14. Conclusion
  15. Limitations
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Evaluation framework using LLMs.
  • Figure 2: Window analysis was performed by selecting any consecutive four systems from the human rankings of the 12 systems ("Base"). For instance, x=4 involves calculating the Pearson correlation ($r$) using the systems ranked from 1st to 4th in the human rankings. In contrast to conventional GEC metrics, which exhibit unstable correlations, GPT-4 demonstrates relatively stable correlations.
  • Figure 3: Prompts used for edit-based evaluation and sentence-based evaluation by LLMs
  • Figure 4: The distribution of scores assigned by LLMs on a 5-point scale. It can be observed that as the LLM scale increases, there is a tendency to assign higher scores (4 or 5 points). Based on our meta-evaluation results indicating better correlation with human judgments as the scale increases, it is suggested that smaller LLMs may underestimate corrections judged to be good by humans.