Measuring Hong Kong Massive Multi-Task Language Understanding

TL;DR

HKMMLU introduces a two-part benchmark to evaluate LLMs on Hong Kong-specific knowledge and Cantonese literacy, combining 26,698 region-focused MCQs across 66 subjects with 90,550 Cantonese–Mandarin translation tasks. Empirical results reveal substantial gaps in current LLM capabilities for Hong Kong content and Cantonese nuances, with DeepSeek-V3 leading zero-shot performance at ~74.8% but still far from universal proficiency. Analyses show that Traditional Chinese performance generally surpasses Simplified, CoT prompting can boost STEM tasks, but long input and reasoning lengths degrade accuracy, and few-shot prompting yields inconsistent gains. A human evaluation indicates near-parity with top LLMs on HKMMLU overall, yet Cantonese questions remain more challenging for AI, underscoring the need for Hong Kong–focused data and language modeling improvements. HKMMLU aims to catalyze progress in multilingual, cross-cultural AI systems that are better aligned with local knowledge and language practices.

Abstract

Multilingual understanding is crucial for the cross-cultural applicability of Large Language Models (LLMs). However, evaluation benchmarks designed for Hong Kong's unique linguistic landscape, which combines Traditional Chinese script with Cantonese as the spoken form and its cultural context, remain underdeveloped. To address this gap, we introduce HKMMLU, a multi-task language understanding benchmark that evaluates Hong Kong's linguistic competence and socio-cultural knowledge. The HKMMLU includes 26,698 multi-choice questions across 66 subjects, organized into four categories: Science, Technology, Engineering, and Mathematics (STEM), Social Sciences, Humanities, and Other. To evaluate the multilingual understanding ability of LLMs, 90,550 Mandarin-Cantonese translation tasks were additionally included. We conduct comprehensive experiments on GPT-4o, Claude 3.7 Sonnet, and 18 open-source LLMs of varying sizes on HKMMLU. The results show that the best-performing model, DeepSeek-V3, struggles to achieve an accuracy of 75\%, significantly lower than that of MMLU and CMMLU. This performance gap highlights the need to improve LLMs' capabilities in Hong Kong-specific language and knowledge domains. Furthermore, we investigate how question language, model size, prompting strategies, and question and reasoning token lengths affect model performance. We anticipate that HKMMLU will significantly advance the development of LLMs in multilingual and cross-cultural contexts, thereby enabling broader and more impactful applications.

Figures (22)

• Figure 1: Overview of HKMMLU subject (partial) with an illustrative task example and bilingual English translations for clarity.
• Figure 2: Question (include choices) token length distribution of multi-choice questions in HKMMLU.
• Figure 3: Token length distribution of Cantonese and Mandarin sentences of translation tasks in HKMMLU.
• Figure 4: Comparing the two top-performing models.
• Figure 5: Comparison of model performance among HKMMLU, TMMLU+, and CMMLU, with most models achieving their best results on CMMLU.