Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Investigation of Domain Adaptation Ability for Chinese Spelling Check Models

Xi Wang, Ruoqing Zhao, Hongliang Dai, Piji Li

TL;DR

Addressing domain adaptation in Chinese Spelling Check (CSC), this work builds three domain-specific datasets (financial, medical, legal) to evaluate cross-domain performance of CSC models and ChatGPT. It formulates CSC as replacing incorrect characters in $X = {x1, x2, ..., xn}$ to yield $Y = {y1, y2, ..., yn}$ and uses an automatic error-generation pipeline with multiple strategies. The study benchmarks four supervised models and one unsupervised model, finding that unsupervised methods generalize better across domains, while supervised models struggle with domain vocabulary; ChatGPT underperforms in CSC. The work provides a public domain-adaptation benchmark for CSC and highlights the importance of domain-aware pretraining and knowledge integration for robust spelling correction in specialized domains.

Abstract

Chinese Spelling Check (CSC) is a meaningful task in the area of Natural Language Processing (NLP) which aims at detecting spelling errors in Chinese texts and then correcting these errors. However, CSC models are based on pretrained language models, which are trained on a general corpus. Consequently, their performance may drop when confronted with downstream tasks involving domain-specific terms. In this paper, we conduct a thorough evaluation about the domain adaption ability of various typical CSC models by building three new datasets encompassing rich domain-specific terms from the financial, medical, and legal domains. Then we conduct empirical investigations in the corresponding domain-specific test datasets to ascertain the cross-domain adaptation ability of several typical CSC models. We also test the performance of the popular large language model ChatGPT. As shown in our experiments, the performances of the CSC models drop significantly in the new domains.

An Empirical Investigation of Domain Adaptation Ability for Chinese Spelling Check Models

TL;DR

Addressing domain adaptation in Chinese Spelling Check (CSC), this work builds three domain-specific datasets (financial, medical, legal) to evaluate cross-domain performance of CSC models and ChatGPT. It formulates CSC as replacing incorrect characters in to yield and uses an automatic error-generation pipeline with multiple strategies. The study benchmarks four supervised models and one unsupervised model, finding that unsupervised methods generalize better across domains, while supervised models struggle with domain vocabulary; ChatGPT underperforms in CSC. The work provides a public domain-adaptation benchmark for CSC and highlights the importance of domain-aware pretraining and knowledge integration for robust spelling correction in specialized domains.

Abstract

Chinese Spelling Check (CSC) is a meaningful task in the area of Natural Language Processing (NLP) which aims at detecting spelling errors in Chinese texts and then correcting these errors. However, CSC models are based on pretrained language models, which are trained on a general corpus. Consequently, their performance may drop when confronted with downstream tasks involving domain-specific terms. In this paper, we conduct a thorough evaluation about the domain adaption ability of various typical CSC models by building three new datasets encompassing rich domain-specific terms from the financial, medical, and legal domains. Then we conduct empirical investigations in the corresponding domain-specific test datasets to ascertain the cross-domain adaptation ability of several typical CSC models. We also test the performance of the popular large language model ChatGPT. As shown in our experiments, the performances of the CSC models drop significantly in the new domains.
Paper Structure (14 sections, 1 figure, 4 tables)

This paper contains 14 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Task Formulation
  4. Domain-Specific Dataset Constructing
  5. Experimental Settings
  6. Evaluation Metrics
  7. Investigated Models
  8. Results and Discussions
  9. Reproduction of the Typical CSC Models
  10. Cross-Domain Evaluation Performance of the Typical CSC Models
  11. Discussions of Results of the Typical CSC Models
  12. Performance of Large Language Model - ChatGPT
  13. Conclusion
  14. Acknowledgement

Figures (1)

  • Figure 1: The output of three models in different domains. Wrong characters are marked in red, ground truth characters are highlighted in green, and predicted characters are highlighted in blue.