A Formal Framework for Fluency-based Multi-Reference Evaluation in Grammatical Error Correction

TL;DR

This paper introduces a formal, fluency-based framework for evaluating grammatical error correction with multiple valid references, addressing the limitations of traditional edit-based, single-reference metrics. By modeling evaluation as an aggregation over multiple references, it unifies GLEU under four strategies—select-best, simple-average, weighted-average, and merged $n$-grams—and analyzes their theoretical properties and cross-linguistic behavior. Empirical results across Czech, Estonian, Ukrainian, and Chinese data show that multi-reference aggregation captures complementary aspects of fluency and coverage, with merged $n$-grams consistently yielding the strongest recall-oriented gains. The framework provides a principled, language-agnostic approach to quantify linguistic diversity in GEC, facilitating fairer, more interpretable cross-linguistic evaluation and guiding future extensions to edit-based settings.

Abstract

Evaluating grammatical error correction requires metrics that reflect the diversity of valid human corrections rather than privileging a single reference. Existing frameworks, largely edit-based and English-centric, rely on rigid alignments between system and reference edits, limiting their applicability in multilingual and generative settings. This paper introduces a formal framework for \textit{fluency-based multi-reference evaluation}, framing $n$-gram similarity as an aggregation problem over multiple legitimate corrections. Within this formulation, we instantiate GLEU through four aggregation strategies--\textsc{select-best}, \textsc{simple-average}, \textsc{weighted-average}, and \textsc{merged-counts}--and analyze their properties of boundedness, monotonicity, and sensitivity to reference variation. Empirical results on Czech, Estonian, Ukrainian, and Chinese corpora show that these strategies capture complementary aspects of fluency and coverage. The framework unifies multi-reference evaluation into a principled, fluency-oriented approach that incorporates linguistic diversity without penalizing legitimate variation.

Figures (1)

• Figure 1: Incremental multi-reference evaluation on the MuCGEC dataset using corpus-level GLEU. The merged variant steadily increases as additional references are included, while select-best plateaus and average remains lower. This illustrates how reference aggregation enhances recall without sacrificing stability.