Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers

Lisa Anne Hendricks, John Mellor, Rosalia Schneider, Jean-Baptiste Alayrac, Aida Nematzadeh

TL;DR

The paper investigates how pretraining data, attention architecture, and loss functions influence multimodal transformers for zero-shot image retrieval. It evaluates a ViLBERT-like base against a strong baseline across six pretraining datasets, and analyzes multimodal attention types (merged, coattention, asymmetric) and losses (MLM, MRM, ITM, with a contrastive ITM variant). Key findings show that dataset quality and language similarity to the downstream task largely determine performance, while true cross-modal attention is essential for best results; image-only or language-only pretraining is not strictly necessary, and contrastive ITM losses do not universally improve end-to-end multimodal models. These insights guide data curation and architectural design toward robust, generalizable visual-linguistic representations, potentially enabling smaller yet effective multimodal models. The work clarifies when and why multimodal attention and certain losses matter, highlighting practical implications for dataset construction and model design in multimodal learning.

Abstract

Recently multimodal transformer models have gained popularity because their performance on language and vision tasks suggest they learn rich visual-linguistic representations. Focusing on zero-shot image retrieval tasks, we study three important factors which can impact the quality of learned representations: pretraining data, the attention mechanism, and loss functions. By pretraining models on six datasets, we observe that dataset noise and language similarity to our downstream task are important indicators of model performance. Through architectural analysis, we learn that models with a multimodal attention mechanism can outperform deeper models with modality specific attention mechanisms. Finally, we show that successful contrastive losses used in the self-supervised learning literature do not yield similar performance gains when used in multimodal transformers

Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers

TL;DR

The paper investigates how pretraining data, attention architecture, and loss functions influence multimodal transformers for zero-shot image retrieval. It evaluates a ViLBERT-like base against a strong baseline across six pretraining datasets, and analyzes multimodal attention types (merged, coattention, asymmetric) and losses (MLM, MRM, ITM, with a contrastive ITM variant). Key findings show that dataset quality and language similarity to the downstream task largely determine performance, while true cross-modal attention is essential for best results; image-only or language-only pretraining is not strictly necessary, and contrastive ITM losses do not universally improve end-to-end multimodal models. These insights guide data curation and architectural design toward robust, generalizable visual-linguistic representations, potentially enabling smaller yet effective multimodal models. The work clarifies when and why multimodal attention and certain losses matter, highlighting practical implications for dataset construction and model design in multimodal learning.

Abstract

Recently multimodal transformer models have gained popularity because their performance on language and vision tasks suggest they learn rich visual-linguistic representations. Focusing on zero-shot image retrieval tasks, we study three important factors which can impact the quality of learned representations: pretraining data, the attention mechanism, and loss functions. By pretraining models on six datasets, we observe that dataset noise and language similarity to our downstream task are important indicators of model performance. Through architectural analysis, we learn that models with a multimodal attention mechanism can outperform deeper models with modality specific attention mechanisms. Finally, we show that successful contrastive losses used in the self-supervised learning literature do not yield similar performance gains when used in multimodal transformers

Paper Structure

This paper contains 25 sections, 5 equations, 5 figures, 6 tables.

Table of Contents

  1. Multimodal Pretraining
  2. Multimodal Transformers
  3. Multimodal Data Processing
  4. Multimodal Attention
  5. Multimodal Loss Functions
  6. Experimental Setup
  7. Base Multimodal Transformer
  8. The Baseline Model
  9. Pretraining Datasets
  10. Evaluation Tasks
  11. Experimental Results
  12. Comparison to a Baseline
  13. Multimodal Data Preprocessing
  14. Pretraining Datasets.
  15. Language-only Pretraining.
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Tracking queries, keys and values for different attention types described in Sec. \ref{['sec:mmattention']}.
  • Figure 2: Effect of pretraining data. The datasets on X axis are ordered based on their zero-shot Flickr scores. IS:Instance Sampling, DS: Dataset Sampling.
  • Figure 3: Comparing models trained with the MSCOCO and CC datasets. We provide the top-1 ranked retrieved image given an input query sentence on the Flickr val dataset. Correctly retrieved images are framed in green and the incorrect ones in red.
  • Figure 4: Ablation studies on number of layers and heads.
  • Figure 5: Comparing top-1 ranked images retrieved with models trained with the different attention mechanisms on the Flickr dataset. Correctly retrieved images are framed in green and the incorrect ones in red.