Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language steering in latent space to mitigate unintended code-switching

Andrey Goncharov, Nikolai Kondusov, Alexey Zaytsev

TL;DR

The paper tackles unintended code-switching in multilingual LLMs and presents latent-space language steering as a cheap, inference-time solution. By identifying language directions with PCA on parallel translations and applying a per-token projection in late layers, the method suppresses language identity while preserving semantics, with negligible overhead and minimal calibration data. Empirically, language directions concentrate in final layers and are linearly separable, enabling 95-99% language classification accuracy and up to a 42% reduction in next-token distribution divergence (up to 42% for English–Chinese) across Qwen2.5 and Llama-3.2, while maintaining fluency. The approach offers a scalable, non-fine-tuning alternative for production multilingual systems, though it is currently validated on smaller models and a limited language set, suggesting avenues for adaptive steering and broader typology coverage.

Abstract

Multilingual Large Language Models (LLMs) often exhibit unintended code-switching, reducing reliability in downstream tasks. We propose latent-space language steering, a lightweight inference-time method that identifies language directions via PCA on parallel translations and steers token embeddings along these axes to control language identity. Our approach mitigates code-switching while preserving semantics with negligible computational overhead and requires only minimal parallel data for calibration. Empirically, we achieve 95-99\% language classification accuracy using a single principal component and reduce next-token distributional divergence by up to 42% across multiple language pairs on Qwen2.5 and Llama-3.2 models. We further analyze the layer-wise evolution of language representations, revealing that language identity concentrates in final layers with near-perfect linear separability.

Language steering in latent space to mitigate unintended code-switching

TL;DR

The paper tackles unintended code-switching in multilingual LLMs and presents latent-space language steering as a cheap, inference-time solution. By identifying language directions with PCA on parallel translations and applying a per-token projection in late layers, the method suppresses language identity while preserving semantics, with negligible overhead and minimal calibration data. Empirically, language directions concentrate in final layers and are linearly separable, enabling 95-99% language classification accuracy and up to a 42% reduction in next-token distribution divergence (up to 42% for English–Chinese) across Qwen2.5 and Llama-3.2, while maintaining fluency. The approach offers a scalable, non-fine-tuning alternative for production multilingual systems, though it is currently validated on smaller models and a limited language set, suggesting avenues for adaptive steering and broader typology coverage.

Abstract

Multilingual Large Language Models (LLMs) often exhibit unintended code-switching, reducing reliability in downstream tasks. We propose latent-space language steering, a lightweight inference-time method that identifies language directions via PCA on parallel translations and steers token embeddings along these axes to control language identity. Our approach mitigates code-switching while preserving semantics with negligible computational overhead and requires only minimal parallel data for calibration. Empirically, we achieve 95-99\% language classification accuracy using a single principal component and reduce next-token distributional divergence by up to 42% across multiple language pairs on Qwen2.5 and Llama-3.2 models. We further analyze the layer-wise evolution of language representations, revealing that language identity concentrates in final layers with near-perfect linear separability.
Paper Structure (21 sections, 3 equations, 4 figures, 4 tables)

This paper contains 21 sections, 3 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related works
  4. Code-switching and multilingual control.
  5. Language structure in representations.
  6. Inference-time control.
  7. Methods
  8. Language Direction Identification
  9. Latent-Space Language Steering
  10. Language Prediction
  11. Results
  12. Language direction identification
  13. Experimental setup.
  14. Language maps reveal emergent clustering.
  15. Language prediction
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Language steering in hidden space: 1. Collect parallel translations of the same concepts; 2. LLM forward pass to get embeddings by layer; 3. Perform PCA by layer on the merged embeddings.
  • Figure :
  • Figure :
  • Figure :