Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval

Ahmed Masry, Megh Thakkar, Patrice Bechard, Sathwik Tejaswi Madhusudhan, Rabiul Awal, Shambhavi Mishra, Akshay Kalkunte Suresh, Srivatsava Daruru, Enamul Hoque, Spandana Gella, Torsten Scholak, Sai Rajeswar

TL;DR

ColMate tackles the gap in multimodal document retrieval by introducing three synergistic components: Masked OCR Language Modeling (MOLM) to enrich visual-token representations, Self-supervised Masked Contrastive Learning (MaskedCL) for cross-modal alignment without labeled pairs, and TopKSim for robust late interaction suited to visual documents. Through pretraining, self-supervised finetuning, and refined scoring, ColMate achieves state-of-the-art results on ViDoRe V1/V2, notably improving out-of-domain generalization and reducing noise from patch-based tokenization. The work demonstrates that modality-specific pretraining objectives and a balanced, top-K based similarity aggregation yield tangible gains, with MaskedCL offering particular promise in low-resource scenarios. Overall, ColMate advances practical multimodal document retrieval with strong generalization, while remaining efficient enough for real-world deployment.

Abstract

Retrieval-augmented generation has proven practical when models require specialized knowledge or access to the latest data. However, existing methods for multimodal document retrieval often replicate techniques developed for text-only retrieval, whether in how they encode documents, define training objectives, or compute similarity scores. To address these limitations, we present ColMate, a document retrieval model that bridges the gap between multimodal representation learning and document retrieval. ColMate utilizes a novel OCR-based pretraining objective, a self-supervised masked contrastive learning objective, and a late interaction scoring mechanism more relevant to multimodal document structures and visual characteristics. ColMate obtains 3.61% improvements over existing retrieval models on the ViDoRe V2 benchmark, demonstrating stronger generalization to out-of-domain benchmarks.

ColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval

TL;DR

ColMate tackles the gap in multimodal document retrieval by introducing three synergistic components: Masked OCR Language Modeling (MOLM) to enrich visual-token representations, Self-supervised Masked Contrastive Learning (MaskedCL) for cross-modal alignment without labeled pairs, and TopKSim for robust late interaction suited to visual documents. Through pretraining, self-supervised finetuning, and refined scoring, ColMate achieves state-of-the-art results on ViDoRe V1/V2, notably improving out-of-domain generalization and reducing noise from patch-based tokenization. The work demonstrates that modality-specific pretraining objectives and a balanced, top-K based similarity aggregation yield tangible gains, with MaskedCL offering particular promise in low-resource scenarios. Overall, ColMate advances practical multimodal document retrieval with strong generalization, while remaining efficient enough for real-world deployment.

Abstract

Retrieval-augmented generation has proven practical when models require specialized knowledge or access to the latest data. However, existing methods for multimodal document retrieval often replicate techniques developed for text-only retrieval, whether in how they encode documents, define training objectives, or compute similarity scores. To address these limitations, we present ColMate, a document retrieval model that bridges the gap between multimodal representation learning and document retrieval. ColMate utilizes a novel OCR-based pretraining objective, a self-supervised masked contrastive learning objective, and a late interaction scoring mechanism more relevant to multimodal document structures and visual characteristics. ColMate obtains 3.61% improvements over existing retrieval models on the ViDoRe V2 benchmark, demonstrating stronger generalization to out-of-domain benchmarks.

Paper Structure

This paper contains 23 sections, 4 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Retrieval & Late Interaction
  4. Visual Document Understanding & Pretraining
  5. Visually Rich Document Retrieval.
  6. Methodology
  7. Masked OCR Language Modeling (MOLM)
  8. TopKSim
  9. Self-supervised Masked Contrastive Learning (MaskedCL).
  10. Experimental Setup
  11. Datasets & Benchmarks
  12. Modeling
  13. Hyperparameters
  14. Evaluation
  15. Main Results
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Overview of ColMate’s key components. (a) Masked OCR Language Modeling (MOLM) explicitly optimizes visual token representations by predicting masked OCR tokens during pretraining. (b) Masked Contrastive Learning (MaskedCL) enables self-supervised cross-modal alignment between masked text and document features when query–document pairs are unavailable. (c) TopKSim refines the late-interaction mechanism by averaging top-K similarity scores during training to reduce noise from patch-based tokenization in visual documents.
  • Figure 2: Ablation Studies: Impact ofColMate Components on ViDoRe Benchmarks (Average nDCG@5)