Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrieval

Alexander Most, Joseph Winjum, Ayan Biswas, Shawn Jones, Nishath Rajiv Ranasinghe, Dan O'Malley, Manish Bhattarai

TL;DR

This paper evaluates two Retrieval-Augmented Generation (RAG) pipelines for document retrieval: a vision-language model-based approach that embeds document images directly (ColPali) and an OCR-based pipeline that extracts text before embedding. It introduces DocDeg, a degraded-document dataset, and a semantic evaluation benchmark with end-to-end QA metrics to assess downstream performance beyond retrieval. Across four degradation levels, OCR-based RAG with Llama 3.2 generally outperforms ColPali in retrieval and semantic QA, while ColPali offers memory and latency advantages at inference. The findings highlight a practical trade-off between robustness to visual noise and semantic accuracy, informing deployment choices for real-world document retrieval systems.

Abstract

Retrieval-Augmented Generation (RAG) has become a popular technique for enhancing the reliability and utility of Large Language Models (LLMs) by grounding responses in external documents. Traditional RAG systems rely on Optical Character Recognition (OCR) to first process scanned documents into text. However, even state-of-the-art OCRs can introduce errors, especially in degraded or complex documents. Recent vision-language approaches, such as ColPali, propose direct visual embedding of documents, eliminating the need for OCR. This study presents a systematic comparison between a vision-based RAG system (ColPali) and more traditional OCR-based pipelines utilizing Llama 3.2 (90B) and Nougat OCR across varying document qualities. Beyond conventional retrieval accuracy metrics, we introduce a semantic answer evaluation benchmark to assess end-to-end question-answering performance. Our findings indicate that while vision-based RAG performs well on documents it has been fine-tuned on, OCR-based RAG is better able to generalize to unseen documents of varying quality. We highlight the key trade-offs between computational efficiency and semantic accuracy, offering practical guidance for RAG practitioners in selecting between OCR-dependent and vision-based document retrieval systems in production environments.

Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrieval

TL;DR

This paper evaluates two Retrieval-Augmented Generation (RAG) pipelines for document retrieval: a vision-language model-based approach that embeds document images directly (ColPali) and an OCR-based pipeline that extracts text before embedding. It introduces DocDeg, a degraded-document dataset, and a semantic evaluation benchmark with end-to-end QA metrics to assess downstream performance beyond retrieval. Across four degradation levels, OCR-based RAG with Llama 3.2 generally outperforms ColPali in retrieval and semantic QA, while ColPali offers memory and latency advantages at inference. The findings highlight a practical trade-off between robustness to visual noise and semantic accuracy, informing deployment choices for real-world document retrieval systems.

Abstract

Retrieval-Augmented Generation (RAG) has become a popular technique for enhancing the reliability and utility of Large Language Models (LLMs) by grounding responses in external documents. Traditional RAG systems rely on Optical Character Recognition (OCR) to first process scanned documents into text. However, even state-of-the-art OCRs can introduce errors, especially in degraded or complex documents. Recent vision-language approaches, such as ColPali, propose direct visual embedding of documents, eliminating the need for OCR. This study presents a systematic comparison between a vision-based RAG system (ColPali) and more traditional OCR-based pipelines utilizing Llama 3.2 (90B) and Nougat OCR across varying document qualities. Beyond conventional retrieval accuracy metrics, we introduce a semantic answer evaluation benchmark to assess end-to-end question-answering performance. Our findings indicate that while vision-based RAG performs well on documents it has been fine-tuned on, OCR-based RAG is better able to generalize to unseen documents of varying quality. We highlight the key trade-offs between computational efficiency and semantic accuracy, offering practical guidance for RAG practitioners in selecting between OCR-dependent and vision-based document retrieval systems in production environments.
Paper Structure (27 sections, 7 equations, 4 figures, 5 tables)

This paper contains 27 sections, 7 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Vision-Language Models for Document Understanding
  5. OCR Integration with Retrieval-Augmented Generation
  6. Vision-Language Models for Document Retrieval
  7. ColPali and the ViDoRe Benchmark
  8. Comparative Benchmarks for Vision-Language Models
  9. Methodology
  10. System Architectures
  11. VLM-Based RAG Pipeline
  12. OCR-Based RAG Pipeline
  13. DocDeg Dataset
  14. Feature Annotation
  15. Q&A pair generation
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Overview of the experimental pipeline comparing VLM-based and OCR-based RAG systems.
  • Figure 2: Zoomed-in view of documents with different degradation levels
  • Figure 3: Retrieval performance across document quality levels
  • Figure 4: Comparative Analysis of RAG System Capabilities