Can Code-Switched Texts Activate a Knowledge Switch in LLMs? A Case Study on English-Korean Code-Switching

TL;DR

This work investigates whether code-switching between English and Korean can activate language-specific knowledge within English-centric multilingual LLMs. It introduces EnKoQA, a synthetic English-Korean CS QA dataset built under the Matrix Language Frame model, and assesses a broad set of multilingual LLMs across knowledge identification and knowledge leveraging tasks. Results show code-switched questions often outperform English and translated Korean baselines, especially in domains tied to Korean language and culture, and suggest that knowledge activation benefits from both identifying relevant knowledge and grounding reasoning in it. The study highlights code-switching as a promising tool to enhance low-resource language tasks and cultural nuance retention, while outlining limitations and avenues for extending the approach to more languages and pretraining regimes.

Abstract

Recent large language models (LLMs) demonstrate multilingual abilities, yet they are English-centric due to dominance of English in training corpora. The limited resource for low-resource languages remains a crucial challenge. Code-switching (CS), a phenomenon where multilingual speakers alternate between languages in a discourse, can convey subtle cultural and linguistic nuances that can be otherwise lost in translation and elicits language-specific knowledge in human communications. In light of this, we investigate whether code-switching can activate, or identify and leverage knowledge for reasoning when LLMs solve low-resource language tasks. To facilitate the research, we first present EnKoQA, a synthetic English-Korean CS question-answering dataset. We provide comprehensive analysis on a variety of multilingual LLMs by subdividing activation process into knowledge identification and knowledge leveraging. Our results demonstrate that compared to English text, CS can faithfully activate knowledge inside LLMs especially on language-specific domains, suggesting the potential of code-switching on low-resource language tasks.

Figures (8)

• Figure 1: A motivating example of knowledge identification between languages. Compared to a question in English (top), a bilingual speaker can "activate" more relevant knowledge with a question in CS (bottom).
• Figure 2: Human evaluation results on faithfulness (top) and helpfulness (bottom) of knowledge lists identified from CS questions and English questions.
• Figure 3: Human evaluation results on pairwise comparisons between knowledge lists identified from CS questions and English questions.
• Figure 4: LLM-as-a-judge evaluation results on pairwise comparison between knowledge lists identified from CS questions and English questions.
• Figure 5: Radar charts of knowledge leveraging performances on all domains across various multilingual LLMs. Green line is code-switching and dashed gray line is English. We report accuracy for the evaluation metric.