How Well Do LLMs Handle Cantonese? Benchmarking Cantonese Capabilities of Large Language Models

TL;DR

The paper addresses the paucity of Cantonese NLP resources by introducing four Cantonese-centric benchmarks (Yue-TruthfulQA, Yue-GSM8K, Yue-ARC-C, Yue-MMLU) and Yue-TRANS for translation, and by evaluating 35 mainstream LLMs on these tests. It systematically analyzes Cantonese data resources, contrasts small-scale Cantonese NLP methods with Cantonese LLMs, and provides a comprehensive experimentation framework including implementation details, evaluation metrics, and cross-model comparisons. Key findings show Cantonese LLMs underperform relative to Mandarin and English, with performance consistently improving under 5-shot prompts, and that different model families excel on different tasks. The authors discuss data quality and cost considerations, highlight challenges like colloquialism and multilingualism, and propose data-centric and model-centric strategies, including data augmentation and targeted model recommendations, to accelerate Cantonese LLM development and promote broader accessibility for Cantonese speakers.

Abstract

The rapid evolution of large language models (LLMs) has transformed the competitive landscape in natural language processing (NLP), particularly for English and other data-rich languages. However, underrepresented languages like Cantonese, spoken by over 85 million people, face significant development gaps, which is particularly concerning given the economic significance of the Guangdong-Hong Kong-Macau Greater Bay Area, and in substantial Cantonese-speaking populations in places like Singapore and North America. Despite its wide use, Cantonese has scant representation in NLP research, especially compared to other languages from similarly developed regions. To bridge these gaps, we outline current Cantonese NLP methods and introduce new benchmarks designed to evaluate LLM performance in factual generation, mathematical logic, complex reasoning, and general knowledge in Cantonese, which aim to advance open-source Cantonese LLM technology. We also propose future research directions and recommended models to enhance Cantonese LLM development.

Figures (15)

• Figure 1: This is number of publications in the ACL Anthology indexed by languages as of September 2024. Following xiang2024cantonese, we retrieve the publications via searching the language name in either the title or the abstract from the ACL Anthology.
• Figure 2: Overview of the paper: We begin by summarizing approaches from small-scale neural networks in Cantonese, then progress to LLMs (work involving existing Cantonese LLMs). In these LLMs, researchers place a greater emphasis on alignment compared to pre-training. Consequently, we introduce four new benchmarks and a translation datatset to evaluate the Cantonese capabilities of LLMs. We analyze the performance of mainstream LLMs on these benchmarks and, in combination with the inherent challenges of Cantonese itself, identify three insightful research opportunities, and we summarize the models that perform good for each specific task. (Figure \ref{['Oppo']}).
• Figure 3: Examples in Yue-Benchmark.
• Figure 4: a, b, c, d represent the performance of various LLMs on Yue-TruthfulQA, Yue-GSM8K, Yue-ARC-C, and Yue-MMLU, in both 0-shot and 5-shot. e, f, g, h correspond to comparisons of performance between four benchmarks and their English or Mandarin version.i indicates the effectiveness of translating from Mandarin and English into Cantonese (Table \ref{['TruthfulQA_Cant']}, \ref{['TruthfulQA_Cant_all']}, \ref{['TruthfulQA_Can_best']}, \ref{['TruthfulQA_Eng_best']}, \ref{['TruthfulQA_incorrect']}, \ref{['TruthfulQA_incorrect']}, \ref{['TruthfulQA_Eng']}, \ref{['GSM8K_Cant']} ,\ref{['GSM8K_Cant_all']}, \ref{['GSM8K_Eng']}, \ref{['ARC-C_Cant']}, \ref{['ARC-C_Cant_all']}, \ref{['ARC_Eng']}, \ref{['MMLU_Cant']}, \ref{['MMLU_Cant_all']}, \ref{['CMMLU']}, \ref{['TRANS_1']}, \ref{['TRANS_2']} for more results).
• Figure 5: LLMs proficient in handling various tasks.