Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OLA: Output Language Alignment in Code-Switched LLM Interactions

Juhyun Oh, Haneul Yoo, Faiz Ghifari Haznitrama, Alice Oh

TL;DR

Code-switching creates implicit expectations for the output language that current multilingual LLMs struggle to infer, leading to systematic misalignment and non-English defaults. The authors introduce OLA, a benchmark targeting Korean–English code-switching across simple intra-sentential and complex instruction–content settings, and demonstrate that standard reasoning prompts do not resolve the issue. They show a consistent, asymmetric bias toward non-English outputs and reveal surface cues heavily influence language choice. A minimal, targeted alignment approach using Code-Switching Aware DPO with about 1K examples substantially improves language alignment and generalizes across language pairs, underscoring that the problem stems from insufficient alignment rather than fundamental capacity limits.

Abstract

Code-switching, alternating between languages within a conversation, is natural for multilingual users, yet poses fundamental challenges for large language models (LLMs). When a user code-switches in their prompt to an LLM, they typically do not specify the expected language of the LLM response, and thus LLMs must infer the output language from contextual and pragmatic cues. We find that current LLMs systematically fail to align with this expectation, responding in undesired languages even when cues are clear to humans. We introduce OLA, a benchmark to evaluate LLMs' Output Language Alignment in code-switched interactions. OLA focuses on Korean--English code-switching and spans simple intra-sentential mixing to instruction-content mismatches. Even frontier models frequently misinterpret implicit language expectation, exhibiting a bias toward non-English responses. We further show this bias generalizes beyond Korean to Chinese and Indonesian pairs. Models also show instability through mid-response switching and language intrusions. Chain-of-Thought prompting fails to resolve these errors, indicating weak pragmatic reasoning about output language. However, Code-Switching Aware DPO with minimal data (about 1K examples) substantially reduces misalignment, suggesting these failures stem from insufficient alignment rather than fundamental limitations. Our results highlight the need to align multilingual LLMs with users' implicit expectations in real-world code-switched interactions.

OLA: Output Language Alignment in Code-Switched LLM Interactions

TL;DR

Code-switching creates implicit expectations for the output language that current multilingual LLMs struggle to infer, leading to systematic misalignment and non-English defaults. The authors introduce OLA, a benchmark targeting Korean–English code-switching across simple intra-sentential and complex instruction–content settings, and demonstrate that standard reasoning prompts do not resolve the issue. They show a consistent, asymmetric bias toward non-English outputs and reveal surface cues heavily influence language choice. A minimal, targeted alignment approach using Code-Switching Aware DPO with about 1K examples substantially improves language alignment and generalizes across language pairs, underscoring that the problem stems from insufficient alignment rather than fundamental capacity limits.

Abstract

Code-switching, alternating between languages within a conversation, is natural for multilingual users, yet poses fundamental challenges for large language models (LLMs). When a user code-switches in their prompt to an LLM, they typically do not specify the expected language of the LLM response, and thus LLMs must infer the output language from contextual and pragmatic cues. We find that current LLMs systematically fail to align with this expectation, responding in undesired languages even when cues are clear to humans. We introduce OLA, a benchmark to evaluate LLMs' Output Language Alignment in code-switched interactions. OLA focuses on Korean--English code-switching and spans simple intra-sentential mixing to instruction-content mismatches. Even frontier models frequently misinterpret implicit language expectation, exhibiting a bias toward non-English responses. We further show this bias generalizes beyond Korean to Chinese and Indonesian pairs. Models also show instability through mid-response switching and language intrusions. Chain-of-Thought prompting fails to resolve these errors, indicating weak pragmatic reasoning about output language. However, Code-Switching Aware DPO with minimal data (about 1K examples) substantially reduces misalignment, suggesting these failures stem from insufficient alignment rather than fundamental limitations. Our results highlight the need to align multilingual LLMs with users' implicit expectations in real-world code-switched interactions.
Paper Structure (70 sections, 10 figures, 10 tables)

This paper contains 70 sections, 10 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Language Confusion and Output Language Misalignment.
  4. Code-switching in Human--LLM Interactions.
  5. OLA Benchmark
  6. Simple Setting
  7. Data Sources and Generation.
  8. Human Verification.
  9. Complex Setting
  10. Data Sources and Generation.
  11. Human Verification.
  12. Experimental Setup
  13. Models.
  14. Metric.
  15. Main Results
  16. ...and 55 more sections

Figures (10)

  • Figure 1: Illustrative failures under intra- and inter-sentential code-switching. The model understands the task content but misses the user's implicit expectation about the output language, causing avoidable friction (intra) and mis-execution of the request with obvious language expectations (inter).
  • Figure 2: Response-level pass rates (%) on the OLA benchmark. (a) Overall performance in Simple and Complex settings. (b) Breakdown by expected output language.
  • Figure 3: Distribution of output language outcomes and error types in the Simple code-switched setting. Hatched segments indicate output-level misalignment; other patterns show intrusion errors within otherwise correct outputs.
  • Figure 4: Effect of script ratio on output language. X-axis shows the percentage of alphabetic (left) or Hangul (right) characters in the prompt.
  • Figure 5: Effect of final word language on output language. Models are significantly more likely to respond in the language of the prompt's final word (all $p$ < 0.001).
  • ...and 5 more figures