Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval

Fengbin Zhu, Zijing Cai, Yuzhe Wang, Pengyang Shao, Wenjie Wang, Fuli Feng, Richang Hong, Tat-Seng Chua

Abstract

Visual Document Retrieval (VDR) requires representations that capture both fine-grained visual details and global document structure to ensure retrieval efficacy while maintaining computational efficiency. Existing VDR models struggle to balance effectiveness and efficiency when processing high-resolution documents: they often either lose fine-grained information or generate an excessive number of visual tokens, resulting in significant indexing overhead and high retrieval latency. In this work, we rethink the visual encoding mechanism and propose a new X-VisEmb paradigm that progresses from multi-resolution sampling and encoding, through cross-granularity feature fusion, to adaptive representation distillation. A preliminary study validates its feasibility and effectiveness in capturing complementary visual cues at varying scales. Building on the insights, we develop MURE, a novel framework that employs VLMs as a hierarchical multi-resolution encoder, integrates resolution-level Matryoshka representation learning (RMRL) for effective feature fusion, and applies a semantic-aware hierarchical clustering mechanism for visual token compression. Experiments on two widely used VDR benchmarks show that our MURE framework consistently beats strong baselines. Furthermore, it significantly outperforms ColPali with only 50% of its visual token budget.

MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval

Abstract

Visual Document Retrieval (VDR) requires representations that capture both fine-grained visual details and global document structure to ensure retrieval efficacy while maintaining computational efficiency. Existing VDR models struggle to balance effectiveness and efficiency when processing high-resolution documents: they often either lose fine-grained information or generate an excessive number of visual tokens, resulting in significant indexing overhead and high retrieval latency. In this work, we rethink the visual encoding mechanism and propose a new X-VisEmb paradigm that progresses from multi-resolution sampling and encoding, through cross-granularity feature fusion, to adaptive representation distillation. A preliminary study validates its feasibility and effectiveness in capturing complementary visual cues at varying scales. Building on the insights, we develop MURE, a novel framework that employs VLMs as a hierarchical multi-resolution encoder, integrates resolution-level Matryoshka representation learning (RMRL) for effective feature fusion, and applies a semantic-aware hierarchical clustering mechanism for visual token compression. Experiments on two widely used VDR benchmarks show that our MURE framework consistently beats strong baselines. Furthermore, it significantly outperforms ColPali with only 50% of its visual token budget.
Paper Structure (33 sections, 8 equations, 9 figures, 2 tables)

This paper contains 33 sections, 8 equations, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Task Definition
  4. Preliminary Study
  5. MURE Framework
  6. X-VisEmb Paradigm
  7. Multi-Resolution Sampling and Encoding
  8. Cross-Granularity Feature Fusion
  9. Hierarchical Document Representation
  10. Resolution-Level Matryoshka Representation Learning
  11. Model Training
  12. Late Interaction
  13. Granularity-Weighted Matryoshka Training Objective
  14. Semantic-aware Hierarchical Clustering
  15. Experiments
  16. ...and 18 more sections

Figures (9)

  • Figure 1: Comparison between our approach (c) and two existing methods (a) and (b). (d) is a sample that demonstrates the necessity of both coarse- and fine-grained features.
  • Figure 2: An overview of our proposed X-VisEmb paradigm.
  • Figure 3: Preliminary experimental results.
  • Figure 4: The overall architecture of MURE . The $S \times p$ represents the aggregate image patch token budget; $p$ denotes the atomic token sequence generated by the visual encoder from a single image pass; $S$ denotes the number of spatial page divisions.
  • Figure 5: Performance analysis of MURE with varying number of target token on ViDoRe V1 and ViDoRe V2.
  • ...and 4 more figures