Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracy

TL;DR

This work addresses the efficiency-accuracy gap in visual document retrieval by introducing HEAVEN, a two-stage hybrid-vector framework that combines fast single-vector retrieval over Visually-Summarized Pages with selective multi-vector reranking using filtered query tokens. It also presents ViMDoc, the first benchmark that jointly evaluates visually rich, long-context, and multi-document retrieval. Empirically, HEAVEN achieves about 99% of state-of-the-art recall while reducing per-query FLOPs by over 99%, demonstrating a strong efficiency-accuracy trade-off across four benchmarks. The approach is OCR-free and leverages POS-based key token selection to significantly cut computation without sacrificing accuracy, offering a scalable foundation for practical visual document retrieval in large corpora.

Abstract

Retrieval over visually rich documents is essential for tasks such as legal discovery, scientific search, and enterprise knowledge management. Existing approaches fall into two paradigms: single-vector retrieval, which is efficient but coarse, and multi-vector retrieval, which is accurate but computationally expensive. To address this trade-off, we propose HEAVEN, a two-stage hybrid-vector framework. In the first stage, HEAVEN efficiently retrieves candidate pages using a single-vector method over Visually-Summarized Pages (VS-Pages), which assemble representative visual layouts from multiple pages. In the second stage, it reranks candidates with a multi-vector method while filtering query tokens by linguistic importance to reduce redundant computations. To evaluate retrieval systems under realistic conditions, we also introduce ViMDOC, the first benchmark for visually rich, multi-document, and long-document retrieval. Across four benchmarks, HEAVEN attains 99.87% of the Recall@1 performance of multi-vector models on average while reducing per-query computation by 99.82%, achieving efficiency and accuracy. Our code and datasets are available at: https://github.com/juyeonnn/HEAVEN

Figures (14)

• Figure 1: Performance comparison of HEAVEN with (1) single-vector models, (2) multi-vector models, and (3) efficiency-oriented variants of multi-vector models (patch pooling and pruning). HEAVEN yields the best trade-off between efficiency and accuracy on the ViMDoc benchmark. Refer to Section \ref{['sec:exp:efficiency']} for details.
• Figure 2: Single- and multi-vector retrieval frameworks.
• Figure 3: Multi-vector models (e.g., ColQwen2.5) outperform single-vector models (e.g., DSE), with pronounced gaps at fine-grained retrieval (K=1) but much smaller gaps at coarse-grained retrieval (K=200).
• Figure 4: Top: Visually-summarized pages (VS-pages) construction process. Bottom: Example outputs. More examples are provided in Appendix \ref{['app:vs-page-example']}.
• Figure 5: Overall pipeline of HEAVEN. (Stage 1) Efficiently retrieves coarse candidate pages via single-vector retrieval, enhanced with visually-summarized pages (VS-pages). (Stage 2) Refines and reranks these candidates using a multi-vector model, enhanced with filtered key query tokens, for fine-grained retrieval.