Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Centered Masking for Language-Image Pre-Training

Mingliang Liang, Martha Larson

TL;DR

GLIP tackles the high computational cost of Vision-Language pretraining by replacing FLIP's random patch masking with centered Gaussian masking, preserving computational savings while improving downstream performance. It maintains the CLIP-style contrastive objective and requires no reconstruction, yet benefits from prioritizing center image regions during masking. Across CC12M/CC3M and a range of tasks (zero-shot classification, image-text retrieval, linear probing, fine-tuning), GLIP delivers gains over FLIP and approaches CLIP on large-scale data, with robustness to masking width and strong generalization to center-agnostic datasets. These results suggest center-prior masked pretraining as a practical and scalable technique for efficient vision-language learning.

Abstract

We introduce Gaussian masking for Language-Image Pre-Training (GLIP) a novel, straightforward, and effective technique for masking image patches during pre-training of a vision-language model. GLIP builds on Fast Language-Image Pre-Training (FLIP), which randomly masks image patches while training a CLIP model. GLIP replaces random masking with centered masking, that uses a Gaussian distribution and is inspired by the importance of image patches at the center of the image. GLIP retains the same computational savings as FLIP, while improving performance across a range of downstream datasets and tasks, as demonstrated by our experimental results. We show the benefits of GLIP to be easy to obtain, requiring no delicate tuning of the Gaussian, and also applicable to data sets containing images without an obvious center focus.

Centered Masking for Language-Image Pre-Training

TL;DR

GLIP tackles the high computational cost of Vision-Language pretraining by replacing FLIP's random patch masking with centered Gaussian masking, preserving computational savings while improving downstream performance. It maintains the CLIP-style contrastive objective and requires no reconstruction, yet benefits from prioritizing center image regions during masking. Across CC12M/CC3M and a range of tasks (zero-shot classification, image-text retrieval, linear probing, fine-tuning), GLIP delivers gains over FLIP and approaches CLIP on large-scale data, with robustness to masking width and strong generalization to center-agnostic datasets. These results suggest center-prior masked pretraining as a practical and scalable technique for efficient vision-language learning.

Abstract

We introduce Gaussian masking for Language-Image Pre-Training (GLIP) a novel, straightforward, and effective technique for masking image patches during pre-training of a vision-language model. GLIP builds on Fast Language-Image Pre-Training (FLIP), which randomly masks image patches while training a CLIP model. GLIP replaces random masking with centered masking, that uses a Gaussian distribution and is inspired by the importance of image patches at the center of the image. GLIP retains the same computational savings as FLIP, while improving performance across a range of downstream datasets and tasks, as demonstrated by our experimental results. We show the benefits of GLIP to be easy to obtain, requiring no delicate tuning of the Gaussian, and also applicable to data sets containing images without an obvious center focus.
Paper Structure (10 sections, 3 equations, 4 figures, 7 tables)

This paper contains 10 sections, 3 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Vision-Language Models
  4. Relative Importance of Image Regions
  5. Photographer behavior
  6. Method
  7. Experiments
  8. Implementation Details
  9. Evaluation.
  10. Conclusion and Outlook

Figures (4)

  • Figure 1: Images from ImageNet-1K deng2009imagenet (left side) and CC12M changpinyo2021cc12m (right side). A contrast can be seen between random masking (left mask) and centered masking (right mask). Centered masking captures more of the main subject of the image.
  • Figure 2: Our GLIP architecture. Following CLIP CLIP2021radford and FLIP li2023flip, we use contrastive loss to pre-train our model. Different from FLIP, we mask image patches with a Gaussian distribution instead of random masking.
  • Figure 3: Comparison of Random and Gaussian Masking Strategies. Image (a) demonstrates a random masking strategy with uniform masking probability. Images (b), (c), and (d) illustrate Gaussian masking with increasing standard deviations ($\sigma$), showcasing the effect of masking that is focused in the center and gradually spreads to the edges.
  • Figure 4: Zero-shot classification performance on ImageNet-1K with different values of $\sigma$ (cf. Formula \ref{['fig:random_gaussian_mask']}). The models, using ViT-B/16 as the image encoder, were pre-trained on the CC3M dataset for 30 epochs and fine-tuned for one more epoch without masking.