Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Building efficient Routed systems for Retrieval

Ramnath Kumar, Prateek Jain, Cho-Jui Hsieh

TL;DR

Late-interaction retrieval methods like ColBERT achieve high accuracy through token-level interactions but incur non-parallelizable computations. FastLane introduces a learnable routing mechanism that dynamically selects the most informative query views and routes them for retrieval, making late-interaction models compatible with ANNS. The method uses a self-attention module to score views, Gumbel-Softmax reparameterization, and straight-through estimator to enable end-to-end training, achieving up to 30x speedups while maintaining competitive accuracy on MS MARCO and TREC-DL-19. This work enables scalable, multi-view retrieval and lays groundwork for longer-context, multilingual, and multimodal retrieval, while noting memory considerations and directions for future work.

Abstract

Late-interaction retrieval models like ColBERT achieve superior accuracy by enabling token-level interactions, but their computational cost hinders scalability and integration with Approximate Nearest Neighbor Search (ANNS). We introduce FastLane, a novel retrieval framework that dynamically routes queries to their most informative representations, eliminating redundant token comparisons. FastLane employs a learnable routing mechanism optimized alongside the embedding model, leveraging self-attention and differentiable selection to maximize efficiency. Our approach reduces computational complexity by up to 30x while maintaining competitive retrieval performance. By bridging late-interaction models with ANNS, FastLane enables scalable, low-latency retrieval, making it feasible for large-scale applications such as search engines, recommendation systems, and question-answering platforms. This work opens pathways for multi-lingual, multi-modal, and long-context retrieval, pushing the frontier of efficient and adaptive information retrieval.

Towards Building efficient Routed systems for Retrieval

TL;DR

Late-interaction retrieval methods like ColBERT achieve high accuracy through token-level interactions but incur non-parallelizable computations. FastLane introduces a learnable routing mechanism that dynamically selects the most informative query views and routes them for retrieval, making late-interaction models compatible with ANNS. The method uses a self-attention module to score views, Gumbel-Softmax reparameterization, and straight-through estimator to enable end-to-end training, achieving up to 30x speedups while maintaining competitive accuracy on MS MARCO and TREC-DL-19. This work enables scalable, multi-view retrieval and lays groundwork for longer-context, multilingual, and multimodal retrieval, while noting memory considerations and directions for future work.

Abstract

Late-interaction retrieval models like ColBERT achieve superior accuracy by enabling token-level interactions, but their computational cost hinders scalability and integration with Approximate Nearest Neighbor Search (ANNS). We introduce FastLane, a novel retrieval framework that dynamically routes queries to their most informative representations, eliminating redundant token comparisons. FastLane employs a learnable routing mechanism optimized alongside the embedding model, leveraging self-attention and differentiable selection to maximize efficiency. Our approach reduces computational complexity by up to 30x while maintaining competitive retrieval performance. By bridging late-interaction models with ANNS, FastLane enables scalable, low-latency retrieval, making it feasible for large-scale applications such as search engines, recommendation systems, and question-answering platforms. This work opens pathways for multi-lingual, multi-modal, and long-context retrieval, pushing the frontier of efficient and adaptive information retrieval.
Paper Structure (26 sections, 4 equations, 4 figures, 2 tables)

This paper contains 26 sections, 4 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Motivation.
  3. Overview and Evaluation.
  4. Contributions.
  5. Related Work
  6. Background
  7. Problem Definition and Notation.
  8. Overview of ${\rm FastLane}$
  9. Our Proposed Approach
  10. Self-Attention Mechanism
  11. Gumbel Softmax Reparameterization
  12. Straight-Through Estimator
  13. Masking and Regularization
  14. End-to-End learning
  15. Experiments
  16. ...and 11 more sections

Figures (4)

  • Figure 1: The query “What is the cost of Apple?” demonstrates how a single phrase can map to diverse user intents: i.e, from stock prices to fruit depending on context. This underscores the need for fine-grained token-level understanding to disambiguate meaning, especially as conventional NLP preprocessing (e.g., stemming, lemmatization) may obscure critical distinctions by dropping stopwords such as "a", "an", etc.
  • Figure 2: Similarity matrices computed from embeddings generated by a pre-trained ColBERT model on the LoTTE benchmark for both query and document examples. The figure illustrates the token-level similarity matrix for a query, highlighting how a small number of distinct token interactions effectively capture the core semantics of the query.
  • Figure 3: Comparison between ColBERT, and ${\rm FastLane}$ paradigm of models.
  • Figure 4: Relative performance of different models on MS MARCO and TREC DL-19 datasets, showing improvement over the Dual-encoder BERT baseline and normalized between 0 and 1.