Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

\textsc{CantoNLU}: A benchmark for Cantonese natural language understanding

Junghyun Min, York Hay Ng, Sophia Chan, Helena Shunhua Zhao, En-Shiun Annie Lee

TL;DR

The paper tackles the scarcity of Cantonese NLP evaluation frameworks by introducing CantoNLU, a GLUE-like benchmark across seven NLU tasks, including a novel WSD dataset. It evaluates three model paradigms: monolingual Cantonese, Cantonese-adapted Mandarin-transfer models, and Mandarin models without Cantonese adaptation, using data from Cantonese Wikipedia, YueData, and diverse corpora. Key findings show Cantonese-adapted models perform best on semantic tasks, monolingual Cantonese models excel at syntactic tasks, and Mandarin models can be competitive when Cantonese data are scarce, underscoring the potential of cross-lingual transfer and data limitations. The work contributes by releasing datasets, code, and model weights to accelerate Cantonese NLP development.

Abstract

Cantonese, although spoken by millions, remains under-resourced due to policy and diglossia. To address this scarcity of evaluation frameworks for Cantonese, we introduce \textsc{\textbf{CantoNLU}}, a benchmark for Cantonese natural language understanding (NLU). This novel benchmark spans seven tasks covering syntax and semantics, including word sense disambiguation, linguistic acceptability judgment, language detection, natural language inference, sentiment analysis, part-of-speech tagging, and dependency parsing. In addition to the benchmark, we provide model baseline performance across a set of models: a Mandarin model without Cantonese training, two Cantonese-adapted models obtained by continual pre-training a Mandarin model on Cantonese text, and a monolingual Cantonese model trained from scratch. Results show that Cantonese-adapted models perform best overall, while monolingual models perform better on syntactic tasks. Mandarin models remain competitive in certain settings, indicating that direct transfer may be sufficient when Cantonese domain data is scarce. We release all datasets, code, and model weights to facilitate future research in Cantonese NLP.

\textsc{CantoNLU}: A benchmark for Cantonese natural language understanding

TL;DR

The paper tackles the scarcity of Cantonese NLP evaluation frameworks by introducing CantoNLU, a GLUE-like benchmark across seven NLU tasks, including a novel WSD dataset. It evaluates three model paradigms: monolingual Cantonese, Cantonese-adapted Mandarin-transfer models, and Mandarin models without Cantonese adaptation, using data from Cantonese Wikipedia, YueData, and diverse corpora. Key findings show Cantonese-adapted models perform best on semantic tasks, monolingual Cantonese models excel at syntactic tasks, and Mandarin models can be competitive when Cantonese data are scarce, underscoring the potential of cross-lingual transfer and data limitations. The work contributes by releasing datasets, code, and model weights to accelerate Cantonese NLP development.

Abstract

Cantonese, although spoken by millions, remains under-resourced due to policy and diglossia. To address this scarcity of evaluation frameworks for Cantonese, we introduce \textsc{\textbf{CantoNLU}}, a benchmark for Cantonese natural language understanding (NLU). This novel benchmark spans seven tasks covering syntax and semantics, including word sense disambiguation, linguistic acceptability judgment, language detection, natural language inference, sentiment analysis, part-of-speech tagging, and dependency parsing. In addition to the benchmark, we provide model baseline performance across a set of models: a Mandarin model without Cantonese training, two Cantonese-adapted models obtained by continual pre-training a Mandarin model on Cantonese text, and a monolingual Cantonese model trained from scratch. Results show that Cantonese-adapted models perform best overall, while monolingual models perform better on syntactic tasks. Mandarin models remain competitive in certain settings, indicating that direct transfer may be sufficient when Cantonese domain data is scarce. We release all datasets, code, and model weights to facilitate future research in Cantonese NLP.
Paper Structure (25 sections, 2 equations, 1 figure, 3 tables)

This paper contains 25 sections, 2 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Cross-lingual transfer.
  4. Cantonese grammar.
  5. Related Work
  6. Cantonese language resources.
  7. CantoNLU and other Cantonese benchmarks.
  8. Building CantoNLU
  9. Novel WSD dataset via manual compilation.
  10. Novel LAJ dataset adapted from error span dataset.
  11. Novel LD dataset from word-aligning and perturbing parallel corpus.
  12. Machine translated English NLI datasets.
  13. Sentiment analysis dataset from restaurant reviews.
  14. POS and DEPS from Cantonese UD.
  15. Experimental Benchmark Setup
  16. ...and 10 more sections

Figures (1)

  • Figure 1: An overview of the tasks in CantoNLU, and our framework for investigating when and how cross-lingual transfer learning from Mandarin is effective for natural language understanding in Cantonese.