PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining

TL;DR

PyramidCLIP tackles semantic mismatch in web-crawled image-text data by building hierarchical input pyramids for both modalities and enforcing six contrastive objectives that align global, local, and object-level semantics. The method introduces peer-level semantics alignment and cross-level relation alignment, with softened negatives to reduce over-penalization of nearly relevant but non-paired samples. It demonstrates substantial improvements over CLIP at 15M pre-training data and state-of-the-art performance on large-scale data, with better data efficiency across classification, retrieval, and detection tasks. The approach leverages lightweight multi-level representations and a dual-stream architecture to achieve finer-grained cross-modal alignment without extra cross-modal encoders, yielding broad transferability.

Abstract

Large-scale vision-language pre-training has achieved promising results on downstream tasks. Existing methods highly rely on the assumption that the image-text pairs crawled from the Internet are in perfect one-to-one correspondence. However, in real scenarios, this assumption can be difficult to hold: the text description, obtained by crawling the affiliated metadata of the image, often suffers from the semantic mismatch and the mutual compatibility. To address these issues, we introduce PyramidCLIP, which constructs an input pyramid with different semantic levels for each modality, and aligns visual elements and linguistic elements in the form of hierarchy via peer-level semantics alignment and cross-level relation alignment. Furthermore, we soften the loss of negative samples (unpaired samples) so as to weaken the strict constraint during the pre-training stage, thus mitigating the risk of forcing the model to distinguish compatible negative pairs. Experiments on five downstream tasks demonstrate the effectiveness of the proposed PyramidCLIP. In particular, with the same amount of 15 million pre-training image-text pairs, PyramidCLIP exceeds CLIP on ImageNet zero-shot classification top-1 accuracy by 10.6%/13.2%/10.0% with ResNet50/ViT-B32/ViT-B16 based image encoder respectively. When scaling to larger datasets, PyramidCLIP achieves the state-of-the-art results on several downstream tasks. In particular, the results of PyramidCLIP-ResNet50 trained on 143M image-text pairs surpass that of CLIP using 400M data on ImageNet zero-shot classification task, significantly improving the data efficiency of CLIP.

Figures (11)

• Figure 1: Problems in the web-crawled image-text pairs. (a)(b)(c) suffer the semantic mismatch between visual modality and linguistic modality, while (d) shows an example of the mutual compatibility with (a). Note that, in (a) the red caption is redundant; in (b) the image outside the red bounding box is the redundant; in (c) the descriptions for the casts in the red boxes are missing; and in (d) the red caption is compatible with the image of (a).
• Figure 1: Pre-training datasets.
• Figure 2: Overall architecture of the proposed PyramidCLIP which is a dual-stream network. The input elements of visual modelling and linguistic modelling both have three-level semantics. The elements of the two modalities interact through peer-level semantics alignment and cross-level relation alignment.
• Figure 2: Zero-shot(ZS) classification results on ImageNet.
• Figure 3: (a) The schematic of CNN-based image encoder. (b) The schematic of ViT-based image encoder. (c) The structure of LeFF module in ViT.