Iterative Multilingual Spectral Attribute Erasure

TL;DR

IMSAE introduces an iterative, SVD-based framework to identify and erase joint demographic bias subspaces across multiple languages, enabling debiasing with or without target-language data. By partitioning source languages into subsets and applying progressive cross-covariance erasure, IMSAE robustly reduces bias while preserving downstream task utility across BERT, LLaMA, and Mistral models. The authors validate IMSAE on eight languages and five demographic attributes using the new MSEFair benchmark, demonstrating improvements over monolingual and cross-lingual baselines in both standard and zero-shot settings. The work provides a principled mechanism for multilingual fairness and contributes a valuable dataset for evaluating cross-lingual debiasing in diverse linguistic contexts.

Abstract

Multilingual representations embed words with similar meanings to share a common semantic space across languages, creating opportunities to transfer debiasing effects between languages. However, existing methods for debiasing are unable to exploit this opportunity because they operate on individual languages. We present Iterative Multilingual Spectral Attribute Erasure (IMSAE), which identifies and mitigates joint bias subspaces across multiple languages through iterative SVD-based truncation. Evaluating IMSAE across eight languages and five demographic dimensions, we demonstrate its effectiveness in both standard and zero-shot settings, where target language data is unavailable, but linguistically similar languages can be used for debiasing. Our comprehensive experiments across diverse language models (BERT, LLaMA, Mistral) show that IMSAE outperforms traditional monolingual and cross-lingual approaches while maintaining model utility.

Figures (4)

• Figure 1: Visualization of gender bias in mBERT French embeddings using t-SNE (blue, male; red, female). Top left: Original embeddings showing clear gender clustering. Top right and bottom: Results after applying IMSAE, demonstrating effective elimination of gender-based patterns through both monolingual (French-only) and cross-lingual (other languages) debiasing approaches. For additional LLM embedding visualizations across different datasets and models, see Figure \ref{['fig:dataset-lang']}.
• Figure 2: A visualization of IMSAE. A sequence of projections is created using SVD based on the input representations (r.v. $\mathbf{X}$), the guarded attributes (r.v. $\mathbf{Z})$ and a language mask that dictates which languages to use.
• Figure 3: Visualization of Llama embeddings across the Multilingual Hate Speech and Multilingual SEFair datasets. "Projected" denotes the projection of the embeddings against the corresponding $\mathbf{Z}$ variable (using SVD). The projection shows that there is a natural mixture both originally and when projecting the representations against Z. Once we project X onto the direction of the protected attribute, the level of mixture increases, but it remains separable, validating the assumption about mixture of different languages.
• Figure 4: The IMSAE algorithm. IMSAE targets attribute-specific bias through cross-covariance between representations $\mathbf{X}$ and protected attributes $\mathbf{Z}$, using SVD to remove only attribute-correlated directions.