Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReinPool: Reinforcement Learning Pooling Multi-Vector Embeddings for Retrieval System

Sungguk Cha, DongWook Kim, Mintae Kim, Youngsub Han, Byoung-Ki Jeon, Sangyeob Lee

TL;DR

ReinPool tackles the scalability challenge of multi-vector document retrieval by learning a reinforcement learning policy that selectively filters token-level embeddings before pooling into a single vector. Trained with an inverse retrieval objective and $NDCG$-based rewards, the method achieves over $1000\times$ compression with substantial retention of retrieval accuracy on Vidore V2 when applied to vision-language embeddings. The approach consistently outperforms static mean/max pooling by 22–33% in absolute $NDCG@3$, demonstrating that learned vector selection can closely approximate full multi-vector performance while enabling scalable indexing. By combining a lightweight filtering policy with GRPO-based optimization and synthetic-query training, ReinPool provides a practical pathway to deploy high-fidelity multi-vector representations at scale.

Abstract

Multi-vector embedding models have emerged as a powerful paradigm for document retrieval, preserving fine-grained visual and textual details through token-level representations. However, this expressiveness comes at a staggering cost: storing embeddings for every token inflates index sizes by over $1000\times$ compared to single-vector approaches, severely limiting scalability. We introduce \textbf{ReinPool}, a reinforcement learning framework that learns to dynamically filter and pool multi-vector embeddings into compact, retrieval-optimized representations. By training with an inverse retrieval objective and NDCG-based rewards, ReinPool identifies and retains only the most discriminative vectors without requiring manual importance annotations. On the Vidore V2 benchmark across three vision-language embedding models, ReinPool compresses multi-vector representations by $746$--$1249\times$ into single vectors while recovering 76--81\% of full multi-vector retrieval performance. Compared to static mean pooling baselines, ReinPool achieves 22--33\% absolute NDCG@3 improvement, demonstrating that learned selection significantly outperforms heuristic aggregation.

ReinPool: Reinforcement Learning Pooling Multi-Vector Embeddings for Retrieval System

TL;DR

ReinPool tackles the scalability challenge of multi-vector document retrieval by learning a reinforcement learning policy that selectively filters token-level embeddings before pooling into a single vector. Trained with an inverse retrieval objective and -based rewards, the method achieves over compression with substantial retention of retrieval accuracy on Vidore V2 when applied to vision-language embeddings. The approach consistently outperforms static mean/max pooling by 22–33% in absolute , demonstrating that learned vector selection can closely approximate full multi-vector performance while enabling scalable indexing. By combining a lightweight filtering policy with GRPO-based optimization and synthetic-query training, ReinPool provides a practical pathway to deploy high-fidelity multi-vector representations at scale.

Abstract

Multi-vector embedding models have emerged as a powerful paradigm for document retrieval, preserving fine-grained visual and textual details through token-level representations. However, this expressiveness comes at a staggering cost: storing embeddings for every token inflates index sizes by over compared to single-vector approaches, severely limiting scalability. We introduce \textbf{ReinPool}, a reinforcement learning framework that learns to dynamically filter and pool multi-vector embeddings into compact, retrieval-optimized representations. By training with an inverse retrieval objective and NDCG-based rewards, ReinPool identifies and retains only the most discriminative vectors without requiring manual importance annotations. On the Vidore V2 benchmark across three vision-language embedding models, ReinPool compresses multi-vector representations by -- into single vectors while recovering 76--81\% of full multi-vector retrieval performance. Compared to static mean pooling baselines, ReinPool achieves 22--33\% absolute NDCG@3 improvement, demonstrating that learned selection significantly outperforms heuristic aggregation.
Paper Structure (18 sections, 2 equations, 1 figure, 1 table)

This paper contains 18 sections, 2 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Neural Information Retrieval
  4. Efficient Indexing for Multi-Vector Models
  5. Learned Pooling and Selection
  6. Methodology
  7. Problem Formulation
  8. ReinPool Framework
  9. Filtering Policy
  10. Pooling
  11. Training Strategy
  12. Synthetic Query Generation
  13. Pre-computation and Setup
  14. Optimization via Inverse Retrieval Rank
  15. Experiments
  16. ...and 3 more sections

Figures (1)

  • Figure 1: Retrieval Performance vs. Cost. Comparison of ReinPool against Full Multi-Vector and Static Mean Pooling baselines. The x-axis represents the total embedding cost (token length), and the y-axis shows the average NDCG@3 on Vidore V2. Horizontal arrows indicate the massive reduction in retrieval cost; vertical arrows depict the performance gain of ReinPool over static mean pooling.