Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GlobalDoc: A Cross-Modal Vision-Language Framework for Real-World Document Image Retrieval and Classification

Souhail Bakkali, Sanket Biswas, Zuheng Ming, Mickaël Coustaty, Marçal Rusiñol, Oriol Ramos Terrades, Josep Lladós

TL;DR

GlobalDoc is introduced, a cross modal transformer-based architecture pre-trained in a self supervised manner using three novel pretext objective tasks that improves the learning of richer semantic concepts by unifying language and visual representations, resulting in more transferable models.

Abstract

Visual document understanding (VDU) has rapidly advanced with the development of powerful multi-modal language models. However, these models typically require extensive document pre-training data to learn intermediate representations and often suffer a significant performance drop in real-world online industrial settings. A primary issue is their heavy reliance on OCR engines to extract local positional information within document pages, which limits the models' ability to capture global information and hinders their generalizability, flexibility, and robustness. In this paper, we introduce GlobalDoc, a cross-modal transformer-based architecture pre-trained in a self-supervised manner using three novel pretext objective tasks. GlobalDoc improves the learning of richer semantic concepts by unifying language and visual representations, resulting in more transferable models. For proper evaluation, we also propose two novel document-level downstream VDU tasks, Few-Shot Document Image Classification (DIC) and Content-based Document Image Retrieval (DIR), designed to simulate industrial scenarios more closely. Extensive experimentation has been conducted to demonstrate GlobalDoc's effectiveness in practical settings.

GlobalDoc: A Cross-Modal Vision-Language Framework for Real-World Document Image Retrieval and Classification

TL;DR

GlobalDoc is introduced, a cross modal transformer-based architecture pre-trained in a self supervised manner using three novel pretext objective tasks that improves the learning of richer semantic concepts by unifying language and visual representations, resulting in more transferable models.

Abstract

Visual document understanding (VDU) has rapidly advanced with the development of powerful multi-modal language models. However, these models typically require extensive document pre-training data to learn intermediate representations and often suffer a significant performance drop in real-world online industrial settings. A primary issue is their heavy reliance on OCR engines to extract local positional information within document pages, which limits the models' ability to capture global information and hinders their generalizability, flexibility, and robustness. In this paper, we introduce GlobalDoc, a cross-modal transformer-based architecture pre-trained in a self-supervised manner using three novel pretext objective tasks. GlobalDoc improves the learning of richer semantic concepts by unifying language and visual representations, resulting in more transferable models. For proper evaluation, we also propose two novel document-level downstream VDU tasks, Few-Shot Document Image Classification (DIC) and Content-based Document Image Retrieval (DIR), designed to simulate industrial scenarios more closely. Extensive experimentation has been conducted to demonstrate GlobalDoc's effectiveness in practical settings.
Paper Structure (15 sections, 12 equations, 3 figures, 5 tables)

This paper contains 15 sections, 12 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Self-Supervised Pre-Training
  4. Cross-Modal Document Understanding
  5. GlobalDoc
  6. Model Architecture
  7. Pre-training Objectives
  8. Novel Document-level Downstream Tasks
  9. Experiments and Results
  10. Experimental Settings
  11. Fine-Tuning on Cross-Modal Tasks
  12. Task I: Proposed Few-Shot DIC
  13. Task II: Proposed Content-based DIR
  14. Task III: Standard Benchmark VDU Tasks
  15. Conclusion

Figures (3)

  • Figure 1: Novel Document-level Downstream Tasks: We propose a new modification to the "pretrain-then-finetune" paradigm for a "closer-to-real" industrial scenario with offline and online stages.
  • Figure 2: Overview of the proposed GlobalDoc framework with the designed pretext learning objectives.
  • Figure 3: Representative uni-modal and cross-modal retrieval samples of GlobalDoc. Zoom in for better visualization. The first column represents the input query -randomly selected- from the test set of RVL-CDIP. The top-5 retrievals are shown in following columns in order. Red and green borders are used to depict the incorrect and correct classes of retrieved documents respectively. For each task setting, the same input example query is used.