Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders

TL;DR

This work investigates how multilingual LLMs internally represent multiple languages by leveraging Cross-Layer Transcoders (CLTs) to build attribution graphs. It reveals a pivot-language mechanism: languages share representations in middle layers while language-specific decoding emerges in later layers, and demonstrates causal control over output language via targeted interventions on a small set of high-frequency features. The authors systematically vary training data mixtures to study how English dominance influences internal mechanisms, showing robust cross-language generalization under imbalance and tokenization-driven disparities in non-English languages. These findings offer a concrete framework for diagnosing and improving multilingual alignment in LLMs, highlighting tokenization and downstream activation strength as key levers for fairness and cross-language performance.

Abstract

Multilingual Large Language Models (LLMs) can process many languages, yet how they internally represent this diversity remains unclear. Do they form shared multilingual representations with language-specific decoding, and if so, why does performance still favor the dominant training language? To address this, we train a series of LLMs on different mixtures of multilingual data and analyze their internal mechanisms using cross-layer transcoders (CLT) and attribution graphs. Our results provide strong evidence for pivot language representations: the model employs nearly identical representations across languages, while language-specific decoding emerges in later layers. Attribution analyses reveal that decoding relies in part on a small set of high-frequency language features in the final layers, which linearly read out language identity from the first layers in the model. By intervening on these features, we can suppress one language and substitute another in the model's outputs. Finally, we study how the dominant training language influences these mechanisms across attribution graphs and decoding pathways. We argue that understanding this pivot-language mechanism is crucial for improving multilingual alignment in LLMs.

Figures (35)

• Figure 1: Overview of our study. We analyze the multilingual behavior of LLMs using attribution graphs derived from cross-layer transcoders (CLTs)ameisen2025circuit. These graphs provide fine-grained feature representations and capture their interactions more effectively than methods used in prior studies, such as Logit Lens.
• Figure 2: Validation cross-entropy loss curves for models trained on multilingual mixtures with varying proportions of English. Despite extreme imbalance (up to 90% English), the models maintain strong next-token prediction performance across languages.
• Figure 3: Rate-weighted multilingual score $H(f)$ across layers, showing that middle layers consistently form a multilingual space, while early and late layers are more language-specific. This pattern holds across model sizes and training data mixtures.
• Figure 4: Circuits identified across different training data mixtures. Middle layers consistently form multilingual clusters, while early and late layers remain more language-specific. The scores at the top right of each cluster indicate language entropy (higher values correspond to more multilingual clusters). The patterns are stable across mixtures, sentence types, and languages.
• Figure 5: Attribution graphs in French, English, and German for the 20% model pruned at 50% for the prompt "Autumn, Winter, Fall, and Spring are the four". Interventions are displayed and language features highlighted in color with their corresponding percentage of activation on their language.