Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?

Hengyi Feng, Zeang Sheng, Meiyi Qiang, Wentao Zhang

TL;DR

The paper investigates why multimodal large language models (MLLMs) underperform as zero-shot multimodal retrievers despite strong generative capabilities. It introduces sparse autoencoders to decompose MLLM activations into interpretable semantic concepts and defines four metrics to characterize these concepts, uncovering a pronounced textual bias and a heavy emphasis on bridging modalities. The key finding is that the most influential components for similarity act as distractors, degrading retrieval performance, and removing them can significantly improve results, offering a training-free route to better retrieval. The work provides the first interpretable analysis of MLLM representations for retrieval and suggests design directions for future retriever-oriented models.

Abstract

Despite the remarkable success of multimodal large language models (MLLMs) in generative tasks, we observe that they exhibit a counterintuitive deficiency in the zero-shot multimodal retrieval task. In this work, we investigate the underlying mechanisms that hinder MLLMs from serving as effective retrievers. With the help of sparse autoencoders (SAEs), we decompose MLLM output representations into interpretable semantic concepts to probe their intrinsic behavior. Our analysis reveals that the representation space of MLLMs is overwhelmingly dominated by textual semantics; the visual information essential for multimodal retrieval only constitutes a small portion. This imbalance is compounded by the heavy focus of MLLMs on bridging image-text modalities, which facilitates generation but homogenizes embeddings and finally diminishes the discriminative power required for multimodal retrieval. We further discover that the specific feature components that contribute most to the similarity computations for MLLMs are in fact distractors that actively degrade retrieval performance. Overall, our work provides the first in-depth interpretability analysis of MLLM representations in the context of multimodal retrieval and offers possible directions for enhancing the multimodal retrieval capabilities of MLLMs.

Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?

TL;DR

The paper investigates why multimodal large language models (MLLMs) underperform as zero-shot multimodal retrievers despite strong generative capabilities. It introduces sparse autoencoders to decompose MLLM activations into interpretable semantic concepts and defines four metrics to characterize these concepts, uncovering a pronounced textual bias and a heavy emphasis on bridging modalities. The key finding is that the most influential components for similarity act as distractors, degrading retrieval performance, and removing them can significantly improve results, offering a training-free route to better retrieval. The work provides the first interpretable analysis of MLLM representations for retrieval and suggests design directions for future retriever-oriented models.

Abstract

Despite the remarkable success of multimodal large language models (MLLMs) in generative tasks, we observe that they exhibit a counterintuitive deficiency in the zero-shot multimodal retrieval task. In this work, we investigate the underlying mechanisms that hinder MLLMs from serving as effective retrievers. With the help of sparse autoencoders (SAEs), we decompose MLLM output representations into interpretable semantic concepts to probe their intrinsic behavior. Our analysis reveals that the representation space of MLLMs is overwhelmingly dominated by textual semantics; the visual information essential for multimodal retrieval only constitutes a small portion. This imbalance is compounded by the heavy focus of MLLMs on bridging image-text modalities, which facilitates generation but homogenizes embeddings and finally diminishes the discriminative power required for multimodal retrieval. We further discover that the specific feature components that contribute most to the similarity computations for MLLMs are in fact distractors that actively degrade retrieval performance. Overall, our work provides the first in-depth interpretability analysis of MLLM representations in the context of multimodal retrieval and offers possible directions for enhancing the multimodal retrieval capabilities of MLLMs.
Paper Structure (18 sections, 11 equations, 4 figures, 3 tables)

This paper contains 18 sections, 11 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminary: zero-shot multimodal retrieval with MLLMs
  3. What hinders MLLMs from being effective retrievers? A mechanistic analysis with sparse autoencoders
  4. Evaluation metrics of learned concepts
  5. Energy.
  6. Modality Score.
  7. Bridge Score.
  8. Retrieval Attribution Score.
  9. Observation: Textual information dominates representations in MLLMs
  10. Observation: MLLMs concentrate most of their representational efforts on bridging image-text modalities
  11. Observation: Dominant components in similarity computations are counterproductive for retrieval
  12. Related works
  13. Multimodal large language models
  14. Multimodal retrieval
  15. Conclusion
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Multimodal retrieval performance of MLLMs and CLIP on the CIRR $((q_i, q_t) \to c_i)$ dataset. The results illustrate the inferior performance of MLLMs.
  • Figure 2: Distribution of modality scores for learned concepts by (a) CLIP, (b) SigLIP2, (c) Qwen3-VL-8B-Instruct, and (d) Paligemma2-3b-Mix-224. The Modality Score quantifies the bias of each concept towards the image modality (blue region) or the text modality (red region). The distributions are visualized using Kernel Density Estimation (KDE) method KDE based on concept activation statistics.
  • Figure 3: Cumulative energy distribution across concept ranks for different multimodal models. The curves show the percentage of total energy captured as concepts are ranked by their individual energy values.
  • Figure 4: Retrieval performance on the subset (3k queries) of CIRR $((q_i, q_t) \to c_i)$ and OVEN $((q_i, q_t) \to (c_i, c_t))$ datasets. “Base” uses the full input; “w/o image” and “w/o prompt.” denote the removal of image tokens and prompt tokens, respectively.