When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training

TL;DR

The paper investigates how language-agnostic concept spaces emerge during multilingual pretraining and tests their causal role using activation patching (cross-lingual concept patching). It analyzes 26 EuroLLM checkpoints with a curated set of 256 source-target noun concepts drawn from Multi-SimLex across 11 languages, comparing patching against unpatched translations for three target outputs. The findings show that shared concept spaces appear early and are refined during training, but alignment is language-dependent and influenced by training data composition; some translation gains arise from sense disambiguation or copying avoidance rather than genuine translation improvements. These results illuminate the training dynamics of cross-lingual alignment and demonstrate the value of causal interpretability methods in multilingual contexts, especially for guiding data strategy in low-resource settings.

Abstract

Training Large Language Models (LLMs) with high multilingual coverage is becoming increasingly important -- especially when monolingual resources are scarce. Recent studies have found that LLMs process multilingual inputs in shared concept spaces, thought to support generalization and cross-lingual transfer. However, these prior studies often do not use causal methods, lack deeper error analysis or focus on the final model only, leaving open how these spaces emerge during training. We investigate the development of language-agnostic concept spaces during pretraining of EuroLLM through the causal interpretability method of activation patching. We isolate cross-lingual concept representations, then inject them into a translation prompt to investigate how consistently translations can be altered, independently of the language. We find that shared concept spaces emerge early} and continue to refine, but that alignment with them is language-dependent}. Furthermore, in contrast to prior work, our fine-grained manual analysis reveals that some apparent gains in translation quality reflect shifts in behavior -- like selecting senses for polysemous words or translating instead of copying cross-lingual homographs -- rather than improved translation ability. Our findings offer new insight into the training dynamics of cross-lingual alignment and the conditions under which causal interpretability methods offer meaningful insights in multilingual contexts.

Figures (16)

• Figure 1: We extend cross-lingual concept patching dumas-etal-2025-separating through a systematic application over pretraining. The first row shows the vanilla translation of the concept $\textsc{skirt}$ from Estonian $\rightarrow$ French. In the second, we replace the activation with averaged activations from Spanish $\rightarrow$ English and Polish $\rightarrow$ Russian for $\textsc{mill}$. Though the representation comes from a distinct set of languages, it can induce $\textsc{mill}$ in French.
• Figure 2: Mean word-level translation accuracy over checkpoints for source prompts used for patching, grouped by a selection of output languages. The red dotted line indicates the start of phase two of EuroLLM's training.
• Figure 3: Mean word-level translation accuracy over checkpoints under different patching settings for a selection of target language pairs. We overlay en--xx, where xx is zh or ru on es--zh, and fi--ru, zh--ru, respectively. The red dotted line indicates the start of phase two of EuroLLM's training. We show 95% CI over 256 samples.
• Figure 4: Net improvement (in percentage points, capped at $\pm35$pp) of seen over unpatched translation accuracy. Each row shows a target language pair, grouped by input and ordered by output language; each column one checkpoint. The red dotted line indicates the start of phase two of EuroLLM's training.
• Figure 5: Area grid of label distribution for outputs under the seen patching setting for a selection of languages. For ru we show only the 111 concepts for which outputs that were labeled; en and zh were labeled in full. The black dotted line indicates the start of phase two of EuroLLM's training.