DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents

TL;DR

DocumentCLIP introduces an intra-document vision-language pretraining framework that links figures with relevant text across long document sections. It employs salience-aware contrastive learning, a layout Transformer, and novel embeddings (section position and entity cues) to model cross-modal interactions within documents like Wikipedia articles. Pretraining on a large Wikipedia-derived dataset enables strong supervised and zero-shot performance, with ablations confirming the importance of layout information, salience selection, and hard negatives. The approach advances multimodal document understanding with practical benefits for search, accessibility, and reader comprehension in real-world documents.

Abstract

Vision-language pretraining models have achieved great success in supporting multimedia applications by understanding the alignments between images and text. While existing vision-language pretraining models primarily focus on understanding single image associated with a single piece of text, they often ignore the alignment at the intra-document level, consisting of multiple sentences with multiple images. In this work, we propose DocumentCLIP, a salience-aware contrastive learning framework to enforce vision-language pretraining models to comprehend the interaction between images and longer text within documents. Our model is beneficial for the real-world multimodal document understanding like news article, magazines, product descriptions, which contain linguistically and visually richer content. To the best of our knowledge, we are the first to explore multimodal intra-document links by contrastive learning. In addition, we collect a large Wikipedia dataset for pretraining, which provides various topics and structures. Experiments show DocumentCLIP not only outperforms the state-of-the-art baselines in the supervised setting, but also achieves the best zero-shot performance in the wild after human evaluation. Our code is available at https://github.com/FuxiaoLiu/DocumentCLIP.

Figures (5)

• Figure 1: Illustration of our task. Given an article, our model extracts the most relevant text to the image/caption pair. (Left): The document is from our pretraining Wikipedia dataset, which has the section to image/caption pair links as the groundtruth label. We leverage this weak supervision to explore multimodal intra-document interaction by contrastive learning. (Right): An example is from Open Textbook dataset in the wild. The highlights are the most relevant sentences extracted by our model.
• Figure 2: Overview of DocumentCLIP in the training phrase. In this example, $t_{00}, t_{01}, t_{02}$ are the Top 3 sentences selected by the Salient Sentence Extraction from the image/caption pair in yellow. $x^{ic}_{12}$ is the negative query from the image/caption pair in green to generated hard negative pairs. $x^{ic}_{11}$ is the positive query to construct positive pairs and normal negative pairs.
• Figure 3: Distributions of the image number per section and section number per document in our Wikipedia dataset.
• Figure 4: Two examples of Wikipedia articles. The red section is the prediction from baseline method while the blue section is the prediction by DocumentCLIP. The sentence in blue is the most relevant candidate.
• Figure 5: Selected image/caption samples and most relevant sentence predictions from Open Textbook dataset. We compare our predictions with the ones from CLIP, SBert and demonstrate the advance of DocumentCLIP.