Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark

Hao Guo, Xugong Qin, Jun Jie Ou Yang, Peng Zhang, Gangyan Zeng, Yubo Li, Hailun Lin

TL;DR

This work introduces NL-DIR, a natural language–based document image retrieval benchmark designed for fine-grained query semantics over real-world document images. It provides a large-scale dataset (41,795 images with five queries each, totaling 205k queries) and a two-stage retrieval pipeline (recall with cross-modal encoders or OCR-free VDU models, followed by re-ranking with cross-attention) to evaluate zero-shot and fine-tuned performance. The study analyzes a broad range of baselines, reveals key factors in pre-training and alignment, and delivers practical insights on efficiency and robustness, particularly for OCR-free retrieval in challenging conditions. Overall, NL-DIR offers a valuable resource for advancing cross-modal document understanding and building more capable, layout-aware DIR systems.

Abstract

Document image retrieval (DIR) aims to retrieve document images from a gallery according to a given query. Existing DIR methods are primarily based on image queries that retrieve documents within the same coarse semantic category, e.g., newspapers or receipts. However, these methods struggle to effectively retrieve document images in real-world scenarios where textual queries with fine-grained semantics are usually provided. To bridge this gap, we introduce a new Natural Language-based Document Image Retrieval (NL-DIR) benchmark with corresponding evaluation metrics. In this work, natural language descriptions serve as semantically rich queries for the DIR task. The NL-DIR dataset contains 41K authentic document images, each paired with five high-quality, fine-grained semantic queries generated and evaluated through large language models in conjunction with manual verification. We perform zero-shot and fine-tuning evaluations of existing mainstream contrastive vision-language models and OCR-free visual document understanding (VDU) models. A two-stage retrieval method is further investigated for performance improvement while achieving both time and space efficiency. We hope the proposed NL-DIR benchmark can bring new opportunities and facilitate research for the VDU community. Datasets and codes will be publicly available at huggingface.co/datasets/nianbing/NL-DIR.

Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark

TL;DR

This work introduces NL-DIR, a natural language–based document image retrieval benchmark designed for fine-grained query semantics over real-world document images. It provides a large-scale dataset (41,795 images with five queries each, totaling 205k queries) and a two-stage retrieval pipeline (recall with cross-modal encoders or OCR-free VDU models, followed by re-ranking with cross-attention) to evaluate zero-shot and fine-tuned performance. The study analyzes a broad range of baselines, reveals key factors in pre-training and alignment, and delivers practical insights on efficiency and robustness, particularly for OCR-free retrieval in challenging conditions. Overall, NL-DIR offers a valuable resource for advancing cross-modal document understanding and building more capable, layout-aware DIR systems.

Abstract

Document image retrieval (DIR) aims to retrieve document images from a gallery according to a given query. Existing DIR methods are primarily based on image queries that retrieve documents within the same coarse semantic category, e.g., newspapers or receipts. However, these methods struggle to effectively retrieve document images in real-world scenarios where textual queries with fine-grained semantics are usually provided. To bridge this gap, we introduce a new Natural Language-based Document Image Retrieval (NL-DIR) benchmark with corresponding evaluation metrics. In this work, natural language descriptions serve as semantically rich queries for the DIR task. The NL-DIR dataset contains 41K authentic document images, each paired with five high-quality, fine-grained semantic queries generated and evaluated through large language models in conjunction with manual verification. We perform zero-shot and fine-tuning evaluations of existing mainstream contrastive vision-language models and OCR-free visual document understanding (VDU) models. A two-stage retrieval method is further investigated for performance improvement while achieving both time and space efficiency. We hope the proposed NL-DIR benchmark can bring new opportunities and facilitate research for the VDU community. Datasets and codes will be publicly available at huggingface.co/datasets/nianbing/NL-DIR.
Paper Structure (51 sections, 16 figures, 11 tables)

This paper contains 51 sections, 16 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Document Image Retrieval
  4. Visual Document Understanding
  5. Construction of Document Image Datasets
  6. NL-DIR: Natural Language-Based Document Image Retrieval
  7. Dataset Overview
  8. Data Collection
  9. Data Annotation
  10. Methodology
  11. Retrieval Baselines
  12. Contrastive VLMs.
  13. Generative VDU Models.
  14. Evaluation Protocol
  15. Metrics.
  16. ...and 36 more sections

Figures (16)

  • Figure 1: Comparison of different paradigms for DIR tasks. (a): Query-by-image. (b): Query-by-text. (c): Our approach directly retrieves document images using natural language queries.
  • Figure 2: Statistics of NL-DIR. (a) Various types of documents. (b) Queries with a particular length. (c) Query and OCR text overlap situation. Best zoom to view.
  • Figure 3: Examples of various types of document images.
  • Figure 4: Examples of queries associated with different types of document images. Best zoom to view.
  • Figure 5: The pipeline for query generation and filtering.
  • ...and 11 more figures