Evaluating the Capability of Large-scale Language Models on Chinese Grammatical Error Correction Task
Fanyi Qu, Chenming Tang, Yunfang Wu
TL;DR
The paper evaluates large-language models on Chinese grammatical error correction across four CGEC datasets from learners and native exams, using both word- and char-level $F_{\text{0.5}}$ metrics with the $M^2$ scorer and CHERRANT, and investigates prompt design effects. It finds that current LLMs still fall short of state-of-the-art CGEC systems, largely due to over-correction, and that performance is highly sensitive to data distribution; newer and larger general-purpose models tend to outperform specialized reasoning models at lower cost. The study also reveals discrepancies between word- and char-level evaluations and hints at possible data leakage in native-exam data. Overall, the work highlights practical considerations for applying LLMs to CGEC and points to future directions to mitigate over-correction and better exploit LLM capabilities.
Abstract
Large-scale language models (LLMs) has shown remarkable capability in various of Natural Language Processing (NLP) tasks and attracted lots of attention recently. However, some studies indicated that large language models fail to achieve promising result beyond the state-of-the-art models in English grammatical error correction (GEC) tasks. In this report, we aim to explore the how large language models perform on Chinese grammatical error correction tasks and provide guidance for future work. We conduct experiments with 3 different LLMs of different model scale on 4 Chinese GEC dataset. Our experimental results indicate that the performances of LLMs on automatic evaluation metrics falls short of the previous sota models because of the problem of over-correction. Furthermore, we also discover notable variations in the performance of LLMs when evaluated on different data distributions. Our findings demonstrates that further investigation is required for the application of LLMs on Chinese GEC task.