Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias

TL;DR

The paper investigates Information Representation Fairness in long-document embeddings and shows that state-of-the-art encoder-based models exhibit strong first-position biases and language-induced representational skew when embedding multi-segment documents. It introduces a permutation-based evaluation framework and a mixed-language extension to quantify biases, and it analyzes attention patterns in pooling-token models to identify front-loaded attention as a contributing factor. It then proposes an inference-time attention calibration that rebalances attention across document positions, reducing bias without retraining. The authors provide empirical results on a multilingual Wikipedia-based corpus and release the evaluation framework at the linked repository, offering practitioners a toolset to diagnose and mitigate fairness gaps in long-context, multilingual retrieval systems.

Abstract

To be discoverable in an embedding-based search process, each part of a document should be reflected in its embedding representation. To quantify any potential reflection biases, we introduce a permutation-based evaluation framework. With this, we observe that state-of-the-art embedding models exhibit systematic positional and language biases when documents are longer and consist of multiple segments. Specifically, early segments and segments in higher-resource languages like English are over-represented, while later segments and segments in lower-resource languages are marginalized. In our further analysis, we find that the positional bias stems from front-loaded attention distributions in pooling-token embeddings, where early tokens receive more attention. To mitigate this issue, we introduce an inference-time attention calibration method that redistributes attention more evenly across document positions, increasing discoverabiltiy of later segments. Our evaluation framework and attention calibration is available at https://github.com/impresso/fair-sentence-transformers

Figures (17)

• Figure 1: We demonstrate that standard document encoders generate skewed embeddings that underrepresent later segments when processing multi-segment documents, such as newspaper pages. We show that an inference-time attention calibration method yields embeddings that are positionally fairer.
• Figure 2: Monolingual experiment instances $(n, \mathbf{L})$, where $n$ varies across rows, and $\mathbf{L}$ varies across columns. Left y-axes show (i) average representation in the global document embedding (mGTE and jina-v3), and (ii) average information retention (jina-v3) per segment position. (Gray bars) show average token length per segment position.
• Figure 3: Preferential treatment of English segments: later segments in $\mathcal{L}_{later}=en$ (top 2 rows) are better represented in the global document embedding (mGTE and jina-v3) and exhibit higher information retention (jina-v3) than later segments in $\mathcal{L}_{later}=de$ (bottom 2 rows).
• Figure 4: Front-loaded self-attention distribution of the $\texttt{<s>}$-query token over key baskets (basket size $\mathfrak{B}{=}128$) in English (left) and Hindi (right) documents ($n{=}3$). Average of the last six transformer layers.
• Figure 5: Monolingual document experiments using attention calibration on mGTE. Notation: 128---$<$s$>$---last_half: Calibrated embeddings with hyperparameters $\mathfrak{B}{=}128$, $\mathfrak{L}^C{=}\{7,\ldots,12\}$