Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Large Language Models Understand, Reason About, and Generate Code-Switched Text?

Genta Indra Winata, David Anugraha, Patrick Amadeus Irawan, Anirban Das, Haneul Yoo, Paresh Dashore, Shreyas Kulkarni, Ruochen Zhang, Haruki Sakajo, Frederikus Hudi, Anaelia Ovalle, Syrielle Montariol, Felix Gaschi, Michael Anugraha, Rutuj Ravindra Puranik, Zawad Hayat Ahmed, Adril Putra Merin, Emmanuele Chersoni

TL;DR

This work introduces CodeMixQA, a comprehensive benchmark for evaluating large language models on code-switched text, including 16 language-pair variations and transliterated forms. It systematically studies understanding, reasoning, and generation under code-switching using three generation strategies: Random Switching, Selective Switching, and Grammar Forcing, and introduces two metrics, Code-Mixing Index (CMI) and Switch-Point Fraction (SPF). The authors analyze a broad set of models and include an automatic evaluation judge, revealing an average performance drop of approximately 11% under code-switching and a tradeoff between naturalness and semantic fidelity in generated outputs. The results highlight persistent challenges in reasoning and generation under multilingual mixing and offer actionable insights for building more robust, linguistically aware multilingual LLMs, with CodeMixQA released as open-source to spur further research.

Abstract

Code-switching is a pervasive phenomenon in multilingual communication, yet the robustness of large language models (LLMs) in mixed-language settings remains insufficiently understood. In this work, we present a comprehensive evaluation of LLM capabilities in understanding, reasoning over, and generating code-switched text. We introduce CodeMixQA a novel benchmark with high-quality human annotations, comprising 16 diverse parallel code-switched language-pair variants that span multiple geographic regions and code-switching patterns, and include both original scripts and their transliterated forms. Using this benchmark, we analyze the reasoning behavior of LLMs on code-switched question-answering tasks, shedding light on how models process and reason over mixed-language inputs. We further conduct a systematic evaluation of LLM-generated synthetic code-switched text, focusing on both naturalness and semantic fidelity, and uncover key limitations in current generation capabilities. Our findings reveal persistent challenges in both reasoning and generation under code-switching conditions and provide actionable insights for building more robust multilingual LLMs. We release the dataset and code as open source.

Can Large Language Models Understand, Reason About, and Generate Code-Switched Text?

TL;DR

This work introduces CodeMixQA, a comprehensive benchmark for evaluating large language models on code-switched text, including 16 language-pair variations and transliterated forms. It systematically studies understanding, reasoning, and generation under code-switching using three generation strategies: Random Switching, Selective Switching, and Grammar Forcing, and introduces two metrics, Code-Mixing Index (CMI) and Switch-Point Fraction (SPF). The authors analyze a broad set of models and include an automatic evaluation judge, revealing an average performance drop of approximately 11% under code-switching and a tradeoff between naturalness and semantic fidelity in generated outputs. The results highlight persistent challenges in reasoning and generation under multilingual mixing and offer actionable insights for building more robust, linguistically aware multilingual LLMs, with CodeMixQA released as open-source to spur further research.

Abstract

Code-switching is a pervasive phenomenon in multilingual communication, yet the robustness of large language models (LLMs) in mixed-language settings remains insufficiently understood. In this work, we present a comprehensive evaluation of LLM capabilities in understanding, reasoning over, and generating code-switched text. We introduce CodeMixQA a novel benchmark with high-quality human annotations, comprising 16 diverse parallel code-switched language-pair variants that span multiple geographic regions and code-switching patterns, and include both original scripts and their transliterated forms. Using this benchmark, we analyze the reasoning behavior of LLMs on code-switched question-answering tasks, shedding light on how models process and reason over mixed-language inputs. We further conduct a systematic evaluation of LLM-generated synthetic code-switched text, focusing on both naturalness and semantic fidelity, and uncover key limitations in current generation capabilities. Our findings reveal persistent challenges in both reasoning and generation under code-switching conditions and provide actionable insights for building more robust multilingual LLMs. We release the dataset and code as open source.
Paper Structure (52 sections, 2 equations, 2 figures, 12 tables)

This paper contains 52 sections, 2 equations, 2 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. CodeMixQA
  4. Design Consideration
  5. Transliteration and Evaluation.
  6. Open-Source.
  7. Data Creation
  8. Random Switching.
  9. Selective Switching.
  10. Grammar Forcing.
  11. Experiments
  12. Tasks and Setup
  13. Understanding.
  14. Reasoning.
  15. Generation.
  16. ...and 37 more sections

Figures (2)

  • Figure 1: Example of code-switching generation for Korean-English using different synthetic data methods. It also presents the human evaluation results leveraging the rubrics, where annotators showed a preference for Grammar Force (Tgt).
  • Figure 2: Reasoning error analysis of Qwen3 30B A3B using GPT-OSS 120B as LLM-as-a-judge across different subsets and languages of $\textcolor{black}{CodeMixQA}$. The analysis follows the error categorization of ovalle2025beg. We observe that errors are more frequent on non-English inputs, whereas transliterated text exhibits fewer errors.