Assessing the Efficacy of Grammar Error Correction: A Human Evaluation Approach in the Japanese Context

TL;DR

This work evaluates the SeqTagger GEC system on English writings by Japanese L1 university students, using ERRANT-based automatic annotation and a human benchmark. It introduces a human-annotated evaluation subset to assess error detection and compares human and model corrections to reveal agreement and disagreement patterns. Key findings show high precision in detection but relatively low recall, with particular difficulty in determiners/articles and context-dependent errors such as tense and noun number. The study provides actionable insights for improving GEC in L1 Japanese learners and releases a dataset to support future research, underscoring the practical impact for classroom feedback and language learning tools.

Abstract

In this study, we evaluated the performance of the state-of-the-art sequence tagging grammar error detection and correction model (SeqTagger) using Japanese university students' writing samples. With an automatic annotation toolkit, ERRANT, we first evaluated SeqTagger's performance on error correction with human expert correction as the benchmark. Then a human-annotated approach was adopted to evaluate Seqtagger's performance in error detection using a subset of the writing dataset. Results indicated a precision of 63.66% and a recall of 20.19% for error correction in the full dataset. For the subset, after manual exclusion of irrelevant errors such as semantic and mechanical ones, the model shows an adjusted precision of 97.98% and an adjusted recall of 42.98% for error detection, indicating the model's high accuracy but also its conservativeness. Thematic analysis on errors undetected by the model revealed that determiners and articles, especially the latter, were predominant. Specifically, in terms of context-independent errors, the model occasionally overlooked basic ones and faced challenges with overly erroneous or complex structures. Meanwhile, context-dependent errors, notably those related to tense and noun number, as well as those possibly influenced by the students' first language (L1), remained particularly challenging.