Developing and Utilizing a Large-Scale Cantonese Dataset for Multi-Tasking in Large Language Models

TL;DR

This work tackles Cantonese as a low-resource NLP setting by constructing YueData, a Cantonese pretraining corpus exceeding 2B tokens drawn from diverse sources such as Hong Kong forums, Apple Daily, Wikipedia zh_yue, and Common Crawl. A rigorous multi-stage pipeline—language filtering, quality/ content filtering, PII masking, and MinHash/LSH deduplication—produces a high-quality dataset used for pretraining YueTung, a 7B Cantonese LLM built atop Qwen-2.5-7b. YueTung achieves state-of-the-art performance on four Cantonese benchmarks and shows robustness across mainstream English and Standard Chinese tasks, indicating strong cross-lingual transfer. The paper highlights the importance of language-specific data quality for improving LLM performance in low-resource languages and demonstrates the practical impact of targeted data construction and supervised fine-tuning.

Abstract

High-quality data resources play a crucial role in learning large language models (LLMs), particularly for low-resource languages like Cantonese. Despite having more than 85 million native speakers, Cantonese is still considered a low-resource language in the field of natural language processing (NLP) due to factors such as the dominance of Mandarin, lack of cohesion within the Cantonese-speaking community, diversity in character encoding and input methods, and the tendency of overseas Cantonese speakers to prefer using English. In addition, rich colloquial vocabulary of Cantonese, English loanwords, and code-switching characteristics add to the complexity of corpus collection and processing. To address these challenges, we collect Cantonese texts from a variety of sources, including open source corpora, Hong Kong-specific forums, Wikipedia, and Common Crawl data. We conduct rigorous data processing through language filtering, quality filtering, content filtering, and de-duplication steps, successfully constructing a high-quality Cantonese corpus of over 2 billion tokens for training large language models. We further refined the model through supervised fine-tuning (SFT) on curated Cantonese tasks, enhancing its ability to handle specific applications. Upon completion of the training, the model achieves state-of-the-art (SOTA) performance on four Cantonese benchmarks. After training on our dataset, the model also exhibits improved performance on other mainstream language tasks.

Figures (2)

• Figure 1: Overview of our work. We construct Cantonese pre-training and SFT data, apply language and quality filters to the former, and derive the latter from it. The base model of YueTung is the Qwen-2.5-7b model, which is trained on the YueData. YueTung achieve SOTA performance on Cantonese benchmarks, and its performance on mainstream language benchmarks not only did not decline but actually improved.
• Figure 2: The results of YueTung-7b and baselines on Yue-Benchmark and mainstream language benchmarks. a and b are YueTung-7b compared with representative LLMs on the Yue-Benchmark (0-shot and 5-shot). c is comparison of YueTung-7b on 0-shot and 5-shot. d is difference between YueTung-7b and Qwen-2.5-7b on the English-GSM8K. e, f, g and h are YueTung-7b compared with base model (Qwen-2.5-7b) on the mainstream language benchmarks (0-shot and 5-shot). The complete results are shown in Table \ref{['TruthfulQA_Cant']}, \ref{['GSM8K_YueTung']}, \ref{['ARC-C_Cant']}, \ref{['MMLU_Cant']}, \ref{['GSM8K_YueTung_all']}, \ref{['ARC-C_Cant_all']}, \ref{['MMLU_Cant_all']}, \ref{['TruthfulQA_Eng']}, \ref{['GSM8K_Eng']}, \ref{['ARC_Eng']}, \ref{['CMMLU']}.