Detection-Correction Structure via General Language Model for Grammatical Error Correction

TL;DR

This paper tackles grammatical error correction (GEC) by introducing DeCoGLM, a GLM-based model that unifies error detection and localized correction within a single framework. Detection uses a fault-tolerant template with a three-label scheme (KEEP, ERROR, INSERT) and a dedicated detection head, while correction leverages autoregressive mask infilling over text pieces, enabling efficient, localized edits. A multi-task objective and attention-mask adjustments foster mutual reinforcement between detection and correction, achieving competitive English and Chinese GEC results with fewer parameters and faster inference than many Seq2Seq baselines. The work also demonstrates that incorporating detection insights can improve LLM-based GEC performance, suggesting a promising direction for integrating detection-correction paradigms into large-scale models.

Abstract

Grammatical error correction (GEC) is a task dedicated to rectifying texts with minimal edits, which can be decoupled into two components: detection and correction. However, previous works have predominantly focused on direct correction, with no prior efforts to integrate both into a single model. Moreover, the exploration of the detection-correction paradigm by large language models (LLMs) remains underdeveloped. This paper introduces an integrated detection-correction structure, named DeCoGLM, based on the General Language Model (GLM). The detection phase employs a fault-tolerant detection template, while the correction phase leverages autoregressive mask infilling for localized error correction. Through the strategic organization of input tokens and modification of attention masks, we facilitate multi-task learning within a single model. Our model demonstrates competitive performance against the state-of-the-art models on English and Chinese GEC datasets. Further experiments present the effectiveness of the detection-correction structure in LLMs, suggesting a promising direction for GEC.

Figures (5)

• Figure 1: Detection and correction process of DeCoGLM. Detection and Correction are incorporated in one General Language Model (GLM).
• Figure 2: The proposed detection-correction structure based on GLM. The example shown above has a source text $\boldsymbol x_s = x_s^1 x_s^2 x_s^3 x_s^4 x_s^5 x_s^6$ and the target text is $\boldsymbol y = x_s^1 c_1^1 x_s^4 x_s^5 c_2^1 c_2^2 x_s^6$. Consistent with GLM, the position IDs and block position IDs are utilized for marking the original positions of text pieces and the inner order of tokens.
• Figure 3: Attention Mask Example. The source text is $\boldsymbol x_s = x_s^1 x_s^2 x_s^3 x_s^4$, and the target text is $\boldsymbol y = c_1^1 x_s^3 x_s^4 c_2^1 c_2^2$. If the cell at row $i$ and column $j$ is colored, it indicates that the $i$ th token can pay attention to the $j$ th token. The region enclosed by dashed lines indicates the attention removed compared to the original GLM.
• Figure 4: Results of detection control on BEA-19 dev set. The heat value represents the value of $F_{0.5}$.
• Figure 5: Loss curves in standard training condition.