Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

C-LLM: Learn to Check Chinese Spelling Errors Character by Character

Kunting Li, Yong Hu, Liang He, Fandong Meng, Jie Zhou

TL;DR

C-LLM is proposed, a Large Language Model-based Chinese Spell Checking method that learns to check errors Character by Character, which enables the model to learn character-level alignment, effectively mitigating issues related to character-level constraints.

Abstract

Chinese Spell Checking (CSC) aims to detect and correct spelling errors in sentences. Despite Large Language Models (LLMs) exhibit robust capabilities and are widely applied in various tasks, their performance on CSC is often unsatisfactory. We find that LLMs fail to meet the Chinese character-level constraints of the CSC task, namely equal length and phonetic similarity, leading to a performance bottleneck. Further analysis reveal that this issue stems from the granularity of tokenization, as current mixed character-word tokenization struggles to satisfy these character-level constraints. To address this issue, we propose C-LLM, a Large Language Model-based Chinese Spell Checking method that learns to check errors Character by Character. Character-level tokenization enables the model to learn character-level alignment, effectively mitigating issues related to character-level constraints. Furthermore, CSC is simplified to replication-dominated and substitution-supplemented tasks. Experiments on two CSC benchmarks demonstrate that C-LLM achieves an average improvement of 10% over existing methods. Specifically, it shows a 2.1% improvement in general scenarios and a significant 12% improvement in vertical domain scenarios, establishing state-of-the-art performance. The source code can be accessed at https://github.com/ktlKTL/C-LLM.

C-LLM: Learn to Check Chinese Spelling Errors Character by Character

TL;DR

C-LLM is proposed, a Large Language Model-based Chinese Spell Checking method that learns to check errors Character by Character, which enables the model to learn character-level alignment, effectively mitigating issues related to character-level constraints.

Abstract

Chinese Spell Checking (CSC) aims to detect and correct spelling errors in sentences. Despite Large Language Models (LLMs) exhibit robust capabilities and are widely applied in various tasks, their performance on CSC is often unsatisfactory. We find that LLMs fail to meet the Chinese character-level constraints of the CSC task, namely equal length and phonetic similarity, leading to a performance bottleneck. Further analysis reveal that this issue stems from the granularity of tokenization, as current mixed character-word tokenization struggles to satisfy these character-level constraints. To address this issue, we propose C-LLM, a Large Language Model-based Chinese Spell Checking method that learns to check errors Character by Character. Character-level tokenization enables the model to learn character-level alignment, effectively mitigating issues related to character-level constraints. Furthermore, CSC is simplified to replication-dominated and substitution-supplemented tasks. Experiments on two CSC benchmarks demonstrate that C-LLM achieves an average improvement of 10% over existing methods. Specifically, it shows a 2.1% improvement in general scenarios and a significant 12% improvement in vertical domain scenarios, establishing state-of-the-art performance. The source code can be accessed at https://github.com/ktlKTL/C-LLM.
Paper Structure (26 sections, 4 equations, 4 figures, 11 tables, 1 algorithm)

This paper contains 26 sections, 4 equations, 4 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Motivation
  4. Problem Formulation
  5. Analysis of LLMs in CSC
  6. Mixed Character-Word Tokenization
  7. Methodology
  8. Character-Level Tokenization
  9. Continued Pre-training
  10. Supervised Fine-tuning
  11. Experiments
  12. Fine-tuning Datasets and Metrics
  13. Baselines
  14. Main Results
  15. Analysis and Discussion
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Encoding differences between the original LLMs and C-LLM.
  • Figure 2: Statistic results of non-homophone characters.
  • Figure 3: Overview of C-LLM. With an LLM (e.g., QWEN bai2023qwen) as the core, the implementation process of C-LLM consists of multiple steps as illustrated in the figure.
  • Figure 4: The trend of character-level correction F1 scores for C-LLM (based on QWEN) across various parameter. Results are presented for both CSCD-NS and LEMON datasets.