Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction

TL;DR

This work tackles the explainability gap in reference-free grammatical error correction metrics by attributing sentence-level scores to individual edits using Shapley values. It formalizes the attribution of the overall score change $\Delta M(H|S)$ across edits, introduces Shapley sampling for scalability, and normalizes attributions for cross-sentence comparability. Empirical results show that edit-level attributions are largely consistent across edit granularities and that larger attribution magnitudes align with human judgments in many cases, achieving roughly 70% agreement. The approach also enables corpus-level analyses, including error-type–based precision and bias inspection, and demonstrates practical utility through case studies and efficiency evaluations, with code available at the provided repository. This contributes a principled, scalable, and interpretable framework for diagnosing and improving sentence-level GEC metrics in both research and application contexts.

Abstract

Various evaluation metrics have been proposed for Grammatical Error Correction (GEC), but many, particularly reference-free metrics, lack explainability. This lack of explainability hinders researchers from analyzing the strengths and weaknesses of GEC models and limits the ability to provide detailed feedback for users. To address this issue, we propose attributing sentence-level scores to individual edits, providing insight into how specific corrections contribute to the overall performance. For the attribution method, we use Shapley values, from cooperative game theory, to compute the contribution of each edit. Experiments with existing sentence-level metrics demonstrate high consistency across different edit granularities and show approximately 70\% alignment with human evaluations. In addition, we analyze biases in the metrics based on the attribution results, revealing trends such as the tendency to ignore orthographic edits. Our implementation is available at \url{https://github.com/naist-nlp/gec-attribute}.

Figures (10)

• Figure 1: The existing metrics are low-explainability.
• Figure 2: Our proposed method improves explainability.
• Figure 4: Cumulative sentences ratio regarding the number of edits. The red line indicates the position where the number of edits is 10.
• Figure 5: CoNLL-2014 results.
• Figure 6: JFLEG results.