Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Look in the Middle: Structural Anchor Pruning for Scalable Visual RAG Indexing

Zhuchenyang Liu, Ziyu Hu, Yao Zhang, Yu Xiao

TL;DR

The paper tackles the index size bottleneck in Visual RAG caused by storing large multi-vector embeddings. It introduces Structural Anchor Pruning (SAP), a training-free, query-agnostic method that identifies semantic structural anchors from the model's middle layers using In-Degree Centrality and a Layer Integration window, and validates it with the Oracle Score Retention (OSR) diagnostic. On ViDoRe, SAP reduces index vectors by over 90% while preserving retrieval fidelity across diverse backbones, revealing a robust Structural Plateau in middle layers and an Alignment Phase in final layers. OSR correlates strongly with downstream NDCG, establishing SAP as a scalable, zero-shot compression technique with practical impact for large-scale Visual RAG deployment.

Abstract

Recent Vision-Language Models (e.g., ColPali) enable fine-grained Visual Document Retrieval (VDR) but incur prohibitive index vector size overheads. Training-free pruning solutions (e.g., EOS-attention based methods) can reduce index vector size by approximately 60% without model adaptation, but often underperform random selection in high-compression scenarios (> 80%). Prior research (e.g., Light-ColPali) attributes this to the conclusion that visual token importance is inherently query-dependent, thereby questioning the feasibility of training-free pruning. In this work, we propose Structural Anchor Pruning (SAP), a training-free pruning method that identifies key visual patches from middle layers to achieve high performance compression. We also introduce Oracle Score Retention (OSR) protocol to evaluate how layer-wise information affects compression efficiency. Evaluations on the ViDoRe benchmark demonstrate that SAP reduces index vectors by over 90% while maintaining robust retrieval fidelity, providing a highly scalable solution for Visual RAG. Furthermore, our OSR-based analysis reveals that semantic structural anchor patches persist in the middle layers, unlike traditional pruning solutions that focus on the final layer where structural signals dissipate.

Look in the Middle: Structural Anchor Pruning for Scalable Visual RAG Indexing

TL;DR

The paper tackles the index size bottleneck in Visual RAG caused by storing large multi-vector embeddings. It introduces Structural Anchor Pruning (SAP), a training-free, query-agnostic method that identifies semantic structural anchors from the model's middle layers using In-Degree Centrality and a Layer Integration window, and validates it with the Oracle Score Retention (OSR) diagnostic. On ViDoRe, SAP reduces index vectors by over 90% while preserving retrieval fidelity across diverse backbones, revealing a robust Structural Plateau in middle layers and an Alignment Phase in final layers. OSR correlates strongly with downstream NDCG, establishing SAP as a scalable, zero-shot compression technique with practical impact for large-scale Visual RAG deployment.

Abstract

Recent Vision-Language Models (e.g., ColPali) enable fine-grained Visual Document Retrieval (VDR) but incur prohibitive index vector size overheads. Training-free pruning solutions (e.g., EOS-attention based methods) can reduce index vector size by approximately 60% without model adaptation, but often underperform random selection in high-compression scenarios (> 80%). Prior research (e.g., Light-ColPali) attributes this to the conclusion that visual token importance is inherently query-dependent, thereby questioning the feasibility of training-free pruning. In this work, we propose Structural Anchor Pruning (SAP), a training-free pruning method that identifies key visual patches from middle layers to achieve high performance compression. We also introduce Oracle Score Retention (OSR) protocol to evaluate how layer-wise information affects compression efficiency. Evaluations on the ViDoRe benchmark demonstrate that SAP reduces index vectors by over 90% while maintaining robust retrieval fidelity, providing a highly scalable solution for Visual RAG. Furthermore, our OSR-based analysis reveals that semantic structural anchor patches persist in the middle layers, unlike traditional pruning solutions that focus on the final layer where structural signals dissipate.
Paper Structure (48 sections, 13 equations, 6 figures, 11 tables)

This paper contains 48 sections, 13 equations, 6 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work and Preliminaries
  3. VDR Multi-Vector Late Interaction
  4. Efficient Visual Document Retrieval
  5. Methodology
  6. Structural Anchor Pruning
  7. Visual In-Degree Centrality.
  8. Head Aggregation.
  9. Layer Integration.
  10. Oracle Score Retention
  11. Evaluation
  12. Evaluation Setup
  13. Diverse VLM Backbones.
  14. Layer Integration Initialization.
  15. Baselines.
  16. ...and 33 more sections

Figures (6)

  • Figure 1: Comparison of Pruning Mechanisms. The left panel illustrates Final Layer EOS Attention, often fails to capture semantic structure. The right panel depicts our Structural Anchor Pruning, which utilizes In-Degree Centrality within the Middle Layers of the LLM backbone. This approach effectively identifies and preserves semantic structural anchor patches.
  • Figure 2: The Mechanics of SAP. We illustrate the Alignment-Aggregation Divergence. Unlike final layers where global signals decay due to MaxSim optimization, the middle layers naturally aggregate information into high-centrality semantic structural anchor patches. SAP exploits this by measuring the In-Degree Centrality between visual tokens to identify key patches without query supervision.
  • Figure 3: Overview of SAP. We compare three pruning paradigms on the ColPali architecture. Left: The shared Vision-Language backbone processes the image. Middle: Conventional methods (Random Selection, Final Layer EOS Attention) fail to identify critical tokens, resulting in lower retention. Right: Our proposed SAP method identifies semantic structural anchor patches via In-Degree Centrality in the model's middle layers, achieving high retrieval performance retention on the ViDoRe v2 benchmark by preserving the document's semantic structure. Bottom: We illustrate the Oracle Score Retention protocol, a white-box diagnostic used to validate our hypothesis. This metric directly compares the MaxSim scores of pruned versus full embeddings, isolating intrinsic information loss from corpus-dependent ranking noise.
  • Figure 4: Efficiency-Fidelity Trade-off. Impact of pruning ratio on NDCG@5 Retention across ColPali, ColQwen2, and JinaEmbeddingsV4 on ViDoRe v2. SAP methods (green) exhibit exceptional stability, significantly outperforming clustering and other pruning baselines at low keep ratios.
  • Figure 5: Oracle Score Retention is a Strong Proxy for Retrieval Performance. We observe a significant positive correlation between intrinsic score preservation and downstream ranking utility. Each data point corresponds to a unique evaluation configuration defined by the model architecture, compression ratio, and dataset subset.
  • ...and 1 more figures