Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models

Xin Li, Yunfei Wu, Xinghua Jiang, Zhihao Guo, Mingming Gong, Haoyu Cao, Yinsong Liu, Deqiang Jiang, Xing Sun

TL;DR

This paper tackles the challenge of fine-grained feature understanding in visual document understanding with LVLMs by identifying a feature collapse when relying on image-level contrastive learning. It introduces Document Object COntrastive learning (DoCo), a two-branch, ROI-based framework that aligns document-object multimodal features with image representations through Intra-DoCo and Inter-DoCo losses, while remaining cost-neutral at inference. DoCo leverages LayoutLMv3-style multimodal features and a novel ROI Aggregation mechanism to produce fine-grained visual cues, and it shows substantial improvements across a suite of VDU benchmarks, narrowing the gap between VDU and general vision-language tasks. The approach is designed as a plug-and-play pre-training method that can be adopted by various LVLMs, and qualitative analyses illustrate improved focus on text-rich regions, with some remaining challenges in document commonsense reasoning and mathematics.

Abstract

Recently, the advent of Large Visual-Language Models (LVLMs) has received increasing attention across various domains, particularly in the field of visual document understanding (VDU). Different from conventional vision-language tasks, VDU is specifically concerned with text-rich scenarios containing abundant document elements. Nevertheless, the importance of fine-grained features remains largely unexplored within the community of LVLMs, leading to suboptimal performance in text-rich scenarios. In this paper, we abbreviate it as the fine-grained feature collapse issue. With the aim of filling this gap, we propose a contrastive learning framework, termed Document Object COntrastive learning (DoCo), specifically tailored for the downstream tasks of VDU. DoCo leverages an auxiliary multimodal encoder to obtain the features of document objects and align them to the visual features generated by the vision encoder of LVLM, which enhances visual representation in text-rich scenarios. It can represent that the contrastive learning between the visual holistic representations and the multimodal fine-grained features of document objects can assist the vision encoder in acquiring more effective visual cues, thereby enhancing the comprehension of text-rich documents in LVLMs. We also demonstrate that the proposed DoCo serves as a plug-and-play pre-training method, which can be employed in the pre-training of various LVLMs without inducing any increase in computational complexity during the inference process. Extensive experimental results on multiple benchmarks of VDU reveal that LVLMs equipped with our proposed DoCo can achieve superior performance and mitigate the gap between VDU and generic vision-language tasks.

Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models

TL;DR

This paper tackles the challenge of fine-grained feature understanding in visual document understanding with LVLMs by identifying a feature collapse when relying on image-level contrastive learning. It introduces Document Object COntrastive learning (DoCo), a two-branch, ROI-based framework that aligns document-object multimodal features with image representations through Intra-DoCo and Inter-DoCo losses, while remaining cost-neutral at inference. DoCo leverages LayoutLMv3-style multimodal features and a novel ROI Aggregation mechanism to produce fine-grained visual cues, and it shows substantial improvements across a suite of VDU benchmarks, narrowing the gap between VDU and general vision-language tasks. The approach is designed as a plug-and-play pre-training method that can be adopted by various LVLMs, and qualitative analyses illustrate improved focus on text-rich regions, with some remaining challenges in document commonsense reasoning and mathematics.

Abstract

Recently, the advent of Large Visual-Language Models (LVLMs) has received increasing attention across various domains, particularly in the field of visual document understanding (VDU). Different from conventional vision-language tasks, VDU is specifically concerned with text-rich scenarios containing abundant document elements. Nevertheless, the importance of fine-grained features remains largely unexplored within the community of LVLMs, leading to suboptimal performance in text-rich scenarios. In this paper, we abbreviate it as the fine-grained feature collapse issue. With the aim of filling this gap, we propose a contrastive learning framework, termed Document Object COntrastive learning (DoCo), specifically tailored for the downstream tasks of VDU. DoCo leverages an auxiliary multimodal encoder to obtain the features of document objects and align them to the visual features generated by the vision encoder of LVLM, which enhances visual representation in text-rich scenarios. It can represent that the contrastive learning between the visual holistic representations and the multimodal fine-grained features of document objects can assist the vision encoder in acquiring more effective visual cues, thereby enhancing the comprehension of text-rich documents in LVLMs. We also demonstrate that the proposed DoCo serves as a plug-and-play pre-training method, which can be employed in the pre-training of various LVLMs without inducing any increase in computational complexity during the inference process. Extensive experimental results on multiple benchmarks of VDU reveal that LVLMs equipped with our proposed DoCo can achieve superior performance and mitigate the gap between VDU and generic vision-language tasks.
Paper Structure (23 sections, 7 equations, 10 figures, 3 tables)

This paper contains 23 sections, 7 equations, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Document Understanding
  4. Large Visual-Language Models
  5. Contrastive Learning
  6. Methodology
  7. Overall Architecture
  8. Document Object Feature Extraction
  9. Document Object Feature Aggregation
  10. Document Object Contrastive Learning
  11. Training Strategies
  12. Experiments
  13. Datasets
  14. Implementation Details
  15. Comparison with State-of-the-arts
  16. ...and 8 more sections

Figures (10)

  • Figure 1: The motivation of the proposed DoCo. (a) Image-level instance discrimination between visual and textual inputs, which aims to learn the holistic representations but fails to extract fine-grained features in text-rich scenarios. (b) Document object discrimination between visual and multimodal inputs, which enhances the visual representation of image encoder in LVLMs and achieves the better visual understanding performance for VDU.
  • Figure 2: A schematic overview of our proposed DoCo. We aim to bridge the representation learning gap between visual features and multimodal features by guiding the former to imitate the features extracted by the latter. Note the branch of multimodal feature extraction is removed after pre-training, which indicates the computational complexity does not increase during the phase of fine-tuning and inference.
  • Figure 3: The proposed ROI Aggregation. The dashed red grid represents the image patch features, and the solid green region denotes the bounding box region. The overlap between each patch and the given region is calculated and serves as the attention mask for the visual aggregation of the document objects. Best viewed in color.
  • Figure 4: An illustrative view of DoCo. The red and purple boxes represent the aggregated visual and multimodal features of document objects, respectively. Best viewed in color.
  • Figure 5: Qualitative results between CLIP ("$\dag$") and DoCo ("$\dag\dag$"). Crucial regions are enlarged for clearer visualization.
  • ...and 5 more figures