LANGSAE EDITING: Improving Multilingual Information Retrieval via Post-hoc Language Identity Removal

TL;DR

The paper tackles language identity bias in multilingual dense retrieval, where embeddings can disproportionately favor same-language evidence. It introduces LangSAE Editing, a post-hoc method that trains an overcomplete sparse autoencoder on pooled embeddings, identifies language-associated latent units via cross-language activation statistics, and suppresses these units at inference with targeted masking before reconstructing embeddings for drop-in cosine scoring. Key contributions include formalizing Language Identity Bias in MLIR, presenting a sparse feature-based masking approach with language-unique and overlapping unit sets, and demonstrating robust gains on Belebele and XQuAD across languages, especially script-distinct ones, with efficient offline and runtime applicability. The method offers practical deployment benefits by enabling retrofitting of existing vector databases without re-encoding raw text and highlights language identity as a manageable, data-efficient signal within embedding spaces that, when suppressed, improves cross-language retrieval quality and coverage.

Abstract

Dense retrieval in multilingual settings often searches over mixed-language collections, yet multilingual embeddings encode language identity alongside semantics. This language signal can inflate similarity for same-language pairs and crowd out relevant evidence written in other languages. We propose LANGSAE EDITING, a post-hoc sparse autoencoder trained on pooled embeddings that enables controllable removal of language-identity signal directly in vector space. The method identifies language-associated latent units using cross-language activation statistics, suppresses these units at inference time, and reconstructs embeddings in the original dimensionality, making it compatible with existing vector databases without retraining the base encoder or re-encoding raw text. Experiments across multiple languages show consistent improvements in ranking quality and cross-language coverage, with especially strong gains for script-distinct languages.

Figures (3)

• Figure 1: Overview of the LangSAE Editing pipeline. Phase I: Train an overcomplete sparse autoencoder on pooled embeddings from a frozen multilingual encoder. Phase II: Compute per-language activation frequencies and select language-associated features to form a mask. Phase III: Encode text, apply the mask in latent space, and decode to obtain an edited embedding for retrieval.
• Figure 2: t-SNE projections of pooled embeddings (multilingual-e5-large, 1000 samples per language). Left: Base encoder embeddings. Middle:Inverse mask embeddings reconstructed using only the language-associated units. Right:LangSAE Edited embeddings after suppressing the language-associated units.
• Figure 3: Sensitivity to $\tau$. Macro nDCG@20 as a function of the activation-frequency threshold $\tau$ (left axis), together with the average number of latent units per language whose activation frequency exceeds $\tau$ (right axis). As $\tau$ decreases slightly below 1.0, the set of frequently active units grows rapidly, which propagates to a much larger suppression set and can trigger over-masking.