MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval

TL;DR

This work addresses modality shortcut in unified multimodal encoders by introducing Modality Composition Awareness (MCA). MCA adds two objectives, Modality Composition Preference (MCP) and Modality Composition Regularization (MCR), to the standard contrastive learning objective, enforcing that composed inputs outperform unimodal parts and remain grounded in a prototype built from unimodal embeddings. Empirical results on in-domain and out-of-domain benchmarks show that MCP and MCR together improve robustness to distribution shifts, with notable gains in OOD retrieval and zero-shot grounding while largely preserving in-domain performance. The approach highlights a practical principle for robust composed multimodal retrieval using Multimodal Large Language Models (MLLMs).

Abstract

Multimodal retrieval, which seeks to retrieve relevant content across modalities such as text or image, supports applications from AI search to contents production. Despite the success of separate-encoder approaches like CLIP align modality-specific embeddings with contrastive learning, recent multimodal large language models (MLLMs) enable a unified encoder that directly processes composed inputs. While flexible and advanced, we identify that unified encoders trained with conventional contrastive learning are prone to learn modality shortcut, leading to poor robustness under distribution shifts. We propose a modality composition awareness framework to mitigate this issue. Concretely, a preference loss enforces multimodal embeddings to outperform their unimodal counterparts, while a composition regularization objective aligns multimodal embeddings with prototypes composed from its unimodal parts. These objectives explicitly model structural relationships between the composed representation and its unimodal counterparts. Experiments on various benchmarks show gains in out-of-distribution retrieval, highlighting modality composition awareness as a effective principle for robust composed multimodal retrieval when utilizing MLLMs as the unified encoder.

Figures (6)

• Figure 1: Illustration of modality shortcut problem in multimodal retrieval. (a) is a conceptual example where the red-highlighted text descriptions are ignored, leading the model to follow a shortcut path, instead of the expected composed representation. (b) and (c) are t-SNE visualizations of composed queries, unimodal queries and targets, that randomly sampled from CIRR dataset, under vanilla CL and our MCA, respectively. tSNE implementation details are introduced in §\ref{['sec:tsne']}.
• Figure 2: Modality Composition Awareness (MCA, §\ref{['sec:ourmethod']}). Unimodal parts are shown as dotted circles when not explicitly modeled. (a) Training with vanilla CL: the composed embedding can be close to the target but still align disproportionately with one unimodal part, leading to modality shortcuts. (b) Training with CL and MCA: the composed embedding is explicitly constrained to be closer to the target than any of its unimodal parts by MCP (§\ref{['sec:mcp']}), and anchored to a compositional prototype mixed (Equation \ref{['eq:mix']}) from unimodal embeddings by MCR (§\ref{['sec:mcr']}) to mitigate modality shortcut.
• Figure 3: A conceptual diagram showing how OOD benchmarks evaluate modality shortcut.
• Figure 4: Convergence analysis of MCA. For a fair comparison, the y-axis range for the loss is set to 0–1, and the range for benchmark performance is consistently set to 20.
• Figure 5: Breakdown of MCA though loss ablation.