PosIR: Position-Aware Heterogeneous Information Retrieval Benchmark

TL;DR

PosIR introduces a position-aware, length-diverse multilingual IR benchmark with 310 datasets across 10 languages and 31 domains to diagnose position bias in dense retrieval. It uses a four-stage pipeline—bilingual corpus construction, position-aware candidate generation with precise reference spans, stringent quality control, and multilingual translation—to enable fine-grained, span-based relevance analysis. Experimental results reveal only moderate alignment with MMTEB for short contexts but substantial divergence for long documents, demonstrate pervasive primacy bias with some recency bias, and show that internal model mechanisms differ, as evidenced by gradient-based saliency patterns. Overall, PosIR provides a scalable, reproducible diagnostic framework to foster robust, position-robust retrieval systems and to study how position interacts with language and domain characteristics.

Abstract

While dense retrieval models have achieved remarkable success, rigorous evaluation of their sensitivity to the position of relevant information (i.e., position bias) remains largely unexplored. Existing benchmarks typically employ position-agnostic relevance labels, conflating the challenge of processing long contexts with the bias against specific evidence locations. To address this challenge, we introduce PosIR (Position-Aware Information Retrieval), a comprehensive benchmark designed to diagnose position bias in diverse retrieval scenarios. PosIR comprises 310 datasets spanning 10 languages and 31 domains, constructed through a rigorous pipeline that ties relevance to precise reference spans, enabling the strict disentanglement of document length from information position. Extensive experiments with 10 state-of-the-art embedding models reveal that: (1) Performance on PosIR in long-context settings correlates poorly with the MMTEB benchmark, exposing limitations in current short-text benchmarks; (2) Position bias is pervasive and intensifies with document length, with most models exhibiting primacy bias while certain models show unexpected recency bias; (3) Gradient-based saliency analysis further uncovers the distinct internal attention mechanisms driving these positional preferences. In summary, PosIR serves as a valuable diagnostic framework to foster the development of position-robust retrieval systems.

Figures (9)

• Figure 1: The four-stage data generation pipeline of PosIR.
• Figure 2: Distribution of normalized reference span positions in the English and Chinese datasets.
• Figure 3: Mean nDCG@10 scores of 10 IR models across 20 relative position bins on the English subset of PosIR.
• Figure 4: Mean nDCG@10 scores of 10 IR models across 20 relative position bins in the French-to-English cross-lingual setting of PosIR. "KaLM-Embedding-12B" denotes KaLM-Embedding-Gemma3-12B-2511.
• Figure 5: Gradient-based saliency maps comparing the internal attention dynamics of Qwen3-Embedding-8B and NV-Embed-v2. The x-axis represents the normalized relative position within a document, and the y-axis shows the normalized L2 norm of the gradients. Shaded regions indicate one standard deviation.