DreamLIP: Language-Image Pre-training with Long Captions

TL;DR

Experimental results on a wide rage of downstream tasks demonstrate the consistent superiority of the method, termed DreamLIP, over previous alternatives, highlighting its fine-grained representational capacity.

Abstract

Language-image pre-training largely relies on how precisely and thoroughly a text describes its paired image. In practice, however, the contents of an image can be so rich that well describing them requires lengthy captions (e.g., with 10 sentences), which are usually missing in existing datasets. Consequently, there are currently no clear evidences on whether and how language-image pre-training could benefit from long captions. To figure this out, we first re-caption 30M images with detailed descriptions using a pre-trained Multi-modality Large Language Model (MLLM), and then study the usage of the resulting captions under a contrastive learning framework. We observe that, each sentence within a long caption is very likely to describe the image partially (e.g., an object). Motivated by this, we propose to dynamically sample sub-captions from the text label to construct multiple positive pairs, and introduce a grouping loss to match the embeddings of each sub-caption with its corresponding local image patches in a self-supervised manner. Experimental results on a wide rage of downstream tasks demonstrate the consistent superiority of our method, termed DreamLIP, over previous alternatives, highlighting its fine-grained representational capacity. It is noteworthy that, on the tasks of image-text retrieval and semantic segmentation, our model trained with 30M image-text pairs achieves on par or even better performance than CLIP trained with 400M pairs. Project page is available at https://zyf0619sjtu.github.io/dream-lip.

Figures (9)

• Figure 1: The richness of an image's content often necessitates long captions for adequate description, with each sentence likely conveying a fragment of the image's entirety. Thanks to the long captions, our DreamLIP trained with 30M image-text pairs achieves on par or even better performance than CLIP trained with 400M pairs on the tasks of image-text retrieval, semantic segmentation, and image understanding in MLLM.
• Figure 2: Illustration of DreamLIP. Firstly, we dynamically sample sub-captions from the text label to construct multiple positive pairs. Then, a global multi-positive contrastive loss is used to align text embeddings of sub-captions and global image embedding. Meanwhile, we introduce a grouping loss to match the embeddings of each sub-caption with its corresponding local image patches in a self-supervised manner.
• Figure 3: Statistics of long captions generated by MLLMs (i.e., InstructBLIP, LLAVA-1.5 and ShareGPT4V).
• Figure 4: Visualization of semantic segmentation and image-text retrieval.
• Figure 5: Visualization for Attention Map. The sub-captions corresponding to the attention maps are split from the generated long captions.