Exploring a Unified Vision-Centric Contrastive Alternatives on Multi-Modal Web Documents

TL;DR

This work tackles the limitations of traditional image–text CLIP models when faced with real-world, interleaved multimodal web documents. It introduces VC^2L, a unified vision-centric framework that renders all inputs as images in pixel space and processes them with a single vision transformer, eliminating OCR and text-tokenization requirements. A snippet-level contrastive objective over consecutive multimodal snippets enables learning from loosely aligned web content, supported by three evaluation benchmarks—AnyCIR, SeqCIR, and CSR—to probe cross-modal retrieval, sequential understanding, and transfer to unseen data. Empirical results show VC^2L achieves competitive or superior performance on zero-shot multi-modal retrieval and text embedding tasks (e.g., M-BEIR, MTEB), validating the value of multimodal web data as a training resource and the scalability of a unified, pixel-space approach for multimodal representation learning.

Abstract

Contrastive vision-language models such as CLIP have demonstrated strong performance across a wide range of multimodal tasks by learning from aligned image-text pairs. However, their ability to handle complex, real-world web documents remains limited, particularly in scenarios where text and images are interleaved, loosely aligned, or embedded in visual form. To address these challenges, we propose Vision-Centric Contrastive Learning (VC2L), a unified framework that models text, images, and their combinations using a single vision transformer. VC2L operates entirely in pixel space by rendering all inputs, whether textual, visual, or combined, as images, thus eliminating the need for OCR, text tokenization, or modality fusion strategy. To capture complex cross-modal relationships in multimodal web documents, VC2L employs a snippet-level contrastive learning objective that aligns consecutive multimodal segments, leveraging the inherent coherence of documents without requiring explicitly paired image-text data. To assess the effectiveness of this approach, we introduce three retrieval benchmarks, AnyCIR, SeqCIR, and CSR, designed to evaluate cross-modal retrieval, fine-grained sequential understanding, and generalization to unseen data, respectively. Empirical results show that VC2L achieves competitive or superior performance compared to CLIP-style models on both the proposed benchmarks and established datasets such as M-BEIR and MTEB. These findings underscore the potential of multimodal web data as a valuable training resource for contrastive learning and illustrate the scalability of a unified, vision-centric approach for multimodal representation learning. Code and models are available at: https://github.com/showlab/VC2L.

Figures (9)

• Figure 1: A comparison of image-caption pairs (Alt-Text) and multimodal documents (Wikipedia).
• Figure 2: VC$^2$L explore an alternative vision-centric paradigm for unified vision-language modeling on interleaved web data. A single vision transformer is used to process any image-text modality from pixels and thereby natively learn a unified representation.
• Figure 3: (a): In AnyCIR, we first sample consecutive snippet pairs from distinct documents and use the former snippet to retrieve the latter one. For each query, all the later snippets are candidates. (b): In SeqCIR, we sequentially retrieve the consecutive snippets in multiple rounds. For each query, all the snippets segmented from sampled documents are candidates while ignoring preceding snippets from the previous round. (c): The positive contrastive pair settings of different baseline models.
• Figure 4: t-SNE visualization of interleaved, text and image snippets embedding on OBELICS.
• Figure 5: Visualization of retrieval results on AnyCIR, SeqCIR, and CSR benchmarks.