Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reproducibility, Replicability, and Insights into Visual Document Retrieval with Late Interaction

Jingfen Qiao, Jia-Huei Ju, Xinyu Ma, Evangelos Kanoulas, Andrew Yates

TL;DR

This work systematically investigates reproducibility, replicability, and insights for Visual Document Retrieval (VDR) centered on ColPali's late-interaction mechanism. By reimplementing ColPali across multiple LVLM backbones and evaluating on the ViDoRe benchmark, it confirms that late interaction substantially boosts retrieval performance, though with higher inference cost. It demonstrates that visual-document embeddings can generalize to OCR-based retrieval and remain robust as index size grows, and it analyzes the interplay between query tokens and image patches, revealing a tendency toward matching visually similar patches rather than exact lexical tokens. The findings offer practical guidance for VDR design, highlighting when and how to leverage token–patch interactions and how visual context influences matching across datasets with varying textual content. Overall, the study provides a rigorous blueprint for assessing reproducibility and replicability in multimodal document retrieval and illuminates paths for improving fine-grained visual-text matching.

Abstract

Visual Document Retrieval (VDR) is an emerging research area that focuses on encoding and retrieving document images directly, bypassing the dependence on Optical Character Recognition (OCR) for document search. A recent advance in VDR was introduced by ColPali, which significantly improved retrieval effectiveness through a late interaction mechanism. ColPali's approach demonstrated substantial performance gains over existing baselines that do not use late interaction on an established benchmark. In this study, we investigate the reproducibility and replicability of VDR methods with and without late interaction mechanisms by systematically evaluating their performance across multiple pre-trained vision-language models. Our findings confirm that late interaction yields considerable improvements in retrieval effectiveness; however, it also introduces computational inefficiencies during inference. Additionally, we examine the adaptability of VDR models to textual inputs and assess their robustness across text-intensive datasets within the proposed benchmark, particularly when scaling the indexing mechanism. Furthermore, our research investigates the specific contributions of late interaction by looking into query-patch matching in the context of visual document retrieval. We find that although query tokens cannot explicitly match image patches as in the text retrieval scenario, they tend to match the patch contains visually similar tokens or their surrounding patches.

Reproducibility, Replicability, and Insights into Visual Document Retrieval with Late Interaction

TL;DR

This work systematically investigates reproducibility, replicability, and insights for Visual Document Retrieval (VDR) centered on ColPali's late-interaction mechanism. By reimplementing ColPali across multiple LVLM backbones and evaluating on the ViDoRe benchmark, it confirms that late interaction substantially boosts retrieval performance, though with higher inference cost. It demonstrates that visual-document embeddings can generalize to OCR-based retrieval and remain robust as index size grows, and it analyzes the interplay between query tokens and image patches, revealing a tendency toward matching visually similar patches rather than exact lexical tokens. The findings offer practical guidance for VDR design, highlighting when and how to leverage token–patch interactions and how visual context influences matching across datasets with varying textual content. Overall, the study provides a rigorous blueprint for assessing reproducibility and replicability in multimodal document retrieval and illuminates paths for improving fine-grained visual-text matching.

Abstract

Visual Document Retrieval (VDR) is an emerging research area that focuses on encoding and retrieving document images directly, bypassing the dependence on Optical Character Recognition (OCR) for document search. A recent advance in VDR was introduced by ColPali, which significantly improved retrieval effectiveness through a late interaction mechanism. ColPali's approach demonstrated substantial performance gains over existing baselines that do not use late interaction on an established benchmark. In this study, we investigate the reproducibility and replicability of VDR methods with and without late interaction mechanisms by systematically evaluating their performance across multiple pre-trained vision-language models. Our findings confirm that late interaction yields considerable improvements in retrieval effectiveness; however, it also introduces computational inefficiencies during inference. Additionally, we examine the adaptability of VDR models to textual inputs and assess their robustness across text-intensive datasets within the proposed benchmark, particularly when scaling the indexing mechanism. Furthermore, our research investigates the specific contributions of late interaction by looking into query-patch matching in the context of visual document retrieval. We find that although query tokens cannot explicitly match image patches as in the text retrieval scenario, they tend to match the patch contains visually similar tokens or their surrounding patches.
Paper Structure (31 sections, 4 equations, 5 figures, 4 tables)

This paper contains 31 sections, 4 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Visual Document Retrieval
  4. Reproduced Method: ColPali
  5. Reproducibility
  6. Experimental Setups
  7. Results
  8. Answer
  9. Experimental Setup
  10. Results
  11. Answer
  12. Replicability
  13. Experimental Setup
  14. Results and Discussion
  15. Answer
  16. ...and 16 more sections

Figures (5)

  • Figure 1: Example of late interaction matching between query and patch tokens; The matching document patch is highlighted by yellow. Query: What Services does Health Team Works Provide?
  • Figure 2: Comparison of retrieval effectiveness across various models when increasing index size for visual document and OCR-based text document retrieval; Blue bar refers to image document indexing; Red bar refers to OCR-based text indexing.
  • Figure 3: The visual feature distribution of datasets in ViDoRe benchmark. See Section \ref{['sec:RQ3.1']} for detail.
  • Figure 4: The $p$-value of significant test of visual feature differences between retrieved and non-retrieved documents.
  • Figure 5: Visualization of semantic matching types across single-vector and multi-vector models. The two images on the right display a sequence of patches generated by ColQwen (top) and ColPali (bottom), respectively. Each patch index corresponds to an index in the confusion matrix.