Positional Bias in Multimodal Embedding Models: Do They Favor the Beginning, the Middle, or the End?

TL;DR

This study reveals pervasive positional bias in multimodal embedding models used for image-text retrieval, showing text encoders favor early positions while image encoders prefer the beginning or both ends. It introduces a framework distinguishing context importance from positional bias and employs perturbation and masking to quantify position effects across multiple CLIP-like models and datasets. The findings indicate that bias persists across caption lengths, model sizes, and encoding schemes, and is modulated by positional encodings, loss functions, and multimodal training. The work underscores the need for bias-aware design and mitigation strategies in vision-language systems to ensure robust cross-modal retrieval and downstream tasks.

Abstract

Positional bias - where models overemphasize certain positions regardless of content - has been shown to negatively impact model performance across various tasks. While recent research has extensively examined positional bias in text generation models, its presence and effects in representation models remain underexplored. Even less is known about such biases in multimodal models. In this work, we investigate positional bias in multimodal representation models, specifically in the context of image-text retrieval. We begin by distinguishing between context importance and positional bias, and then assess the presence and extent of positional bias across different models and datasets. Our experiments demonstrate that positional bias is prevalent in multimodal models, but manifests differently across modalities: text encoders tend to exhibit bias toward the beginning of the input, whereas image encoders show bias at both the beginning and end. Furthermore, we find that this bias arises from, or is amplified by, a combination of factors, including the positional encoding scheme, training loss, context importance, and the nature of using image-text pairs in multimodal training.

Figures (9)

• Figure 1: Top-5 text-to-image retrieval results. The caption is divided into six segments. Each row corresponds to a different position where only Segment 0 (highlighted in dark green) is placed, while the remaining positions are masked. Across the six rows, Segment 0 is shifted through all six possible positions.
• Figure 2: Contextual importance analysis across modalities (image and text). The x-axis shows positions, and colors indicate different number of splits.
• Figure 3: Positional bias analysis of text encoders across multimodal models and datasets. Each figure title follows the format: model name # dataset # metric. The x-axis shows positions, and colors indicate different segments.
• Figure 4: Position bias analysis of image encoders across models and datasets. Each figure title follows the format: model name # dataset # metric. The x-axis shows positions, and colors indicate different segments.
• Figure 5: Classification performance on ImageNet for vision-only and CLIP-based models. We report Top-1 accuracy for ResNet-50 and ViT-B/16, both as standalone vision models and as vision backbones within CLIP.