Beyond Patch Aggregation: 3-Pass Pyramid Indexing for Vision-Enhanced Document Retrieval

TL;DR

VisionRAG addresses the limitations of OCR-based and dense patch-based document retrieval by performing OCR-free, vision-guided indexing on page images. It introduces a three-pass pyramid indexing framework that extracts page summaries, section headers, facts, and visual hotspots, assembling them into 11–17 lightweight vectors per page and fusing signals via Reciprocal Rank Fusion. Across FinanceBench and TAT-DQA, VisionRAG achieves strong retrieval and QA performance while substantially reducing memory and latency compared to patch-based methods, demonstrating a model-agnostic, production-friendly approach. The work highlights the practical benefits of explicit semantic fusion for scalable, multimodal document intelligence in enterprise settings.

Abstract

Document centric RAG pipelines usually begin with OCR, followed by brittle heuristics for chunking, table parsing, and layout reconstruction. These text first workflows are costly to maintain, sensitive to small layout shifts, and often lose the spatial cues that contain the answer. Vision first retrieval has emerged as a strong alternative. By operating directly on page images, systems like ColPali and ColQwen preserve structure and reduce pipeline complexity while achieving strong benchmark performance. However, these late interaction models tie retrieval to a specific vision backbone and require storing hundreds of patch embeddings per page, creating high memory overhead and complicating large scale deployment. We introduce VisionRAG, a multimodal retrieval system that is OCR free and model agnostic. VisionRAG indexes documents directly as images, preserving layout, tables, and spatial cues, and builds semantic vectors without committing to a specific extraction. Our three pass pyramid indexing framework creates vectors using global page summaries, section headers, visual hotspots, and fact level cues. These summaries act as lightweight retrieval surrogates. At query time, VisionRAG retrieves the most relevant pages using the pyramid index, then forwards the raw page image encoded as base64 to a multimodal LLM for final question answering. During retrieval, reciprocal rank fusion integrates signals across the pyramid to produce robust ranking. VisionRAG stores only 17 to 27 vectors per page, matching the efficiency of patch based methods while staying flexible across multimodal encoders. On financial document benchmarks, it achieves 0.8051 accuracy at 10 on FinanceBench and 0.9629 recall at 100 on TAT DQA. These results show that OCR free, summary guided multimodal retrieval is a practical and scalable alternative to traditional text extraction pipelines.

Figures (7)

• Figure 1: Evolution of document retrieval approaches. (Top) OCR-based RAG flattens visual structure, losing layout and table context. (Middle) ColPali adds vision awareness via dense patch embeddings ( 1,024 vectors/page) but at high cost. (Bottom) VisionRAG introduces pyramid indexing with semantic fusion across page, section, and fact levels, achieving similar accuracy with only 12--17 vectors per page and no OCR dependency.
• Figure 2: High-level comparison. Right: ColPali encodes dense patch embeddings with late interaction. Leftt: VisionRAG builds compact multi-level indices and fuses results via Reciprocal Rank Fusion (RRF).
• Figure 3: Model– agnostic evaluation on FinanceBench (K=10, n=148). Our Vision RAG framework maintains strong performance across different vision– language models. Results show accuracy, average tokens processed, and relative performance compared to GPT– 4o baseline.
• Figure 4: Model– agnostic evaluation on TAT– DQA (K=10, n=1,644). Vision RAG maintains consistent retrieval performance across different VLMs, with variations primarily reflecting each model's visual understanding capabilities. Vision RAG maintains consistent retrieval performance across vision–language models.
• Figure 5: FinanceBench results with Vision RAG across different retrieval depths. Metrics include Recall@10 (retrieval coverage), nDCG@10 (ranking quality), Accuracy (answer correctness), average tokens passed to generator, and number of test cases (n=148 after filtering unanswerable questions from original 150).