Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction

Sebastian Hofstätter, Omar Khattab, Sophia Althammer, Mete Sertkan, Allan Hanbury

TL;DR

ColBERTer tackles the tension between neural IR quality and the efficiency/interpretability of traditional methods by introducing a contextualized late-interaction model with enhanced reduction. It fuses two-vector refinement with a Bag Of Whole-Words representation (BOW$^2$), contextualized stopword pruning, and optional lexical matching to dramatically cut storage while preserving or improving effectiveness. Through multi-task, multi-stage training and flexible indexing workflows, ColBERTer achieves up to 2.5x storage reductions and can operate with as little as one dimension per token in Uni-ColBERTer, maintaining strong performance on MS MARCO and TREC-DL and showing robust out-of-domain results. The work also provides a transparent scoring interface mapping to whole words, facilitating interpretability and user-facing explanations in neural IR systems.

Abstract

Recent progress in neural information retrieval has demonstrated large gains in effectiveness, while often sacrificing the efficiency and interpretability of the neural model compared to classical approaches. This paper proposes ColBERTer, a neural retrieval model using contextualized late interaction (ColBERT) with enhanced reduction. Along the effectiveness Pareto frontier, ColBERTer's reductions dramatically lower ColBERT's storage requirements while simultaneously improving the interpretability of its token-matching scores. To this end, ColBERTer fuses single-vector retrieval, multi-vector refinement, and optional lexical matching components into one model. For its multi-vector component, ColBERTer reduces the number of stored vectors per document by learning unique whole-word representations for the terms in each document and learning to identify and remove word representations that are not essential to effective scoring. We employ an explicit multi-task, multi-stage training to facilitate using very small vector dimensions. Results on the MS MARCO and TREC-DL collection show that ColBERTer can reduce the storage footprint by up to 2.5x, while maintaining effectiveness. With just one dimension per token in its smallest setting, ColBERTer achieves index storage parity with the plaintext size, with very strong effectiveness results. Finally, we demonstrate ColBERTer's robustness on seven high-quality out-of-domain collections, yielding statistically significant gains over traditional retrieval baselines.

Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction

TL;DR

ColBERTer tackles the tension between neural IR quality and the efficiency/interpretability of traditional methods by introducing a contextualized late-interaction model with enhanced reduction. It fuses two-vector refinement with a Bag Of Whole-Words representation (BOW), contextualized stopword pruning, and optional lexical matching to dramatically cut storage while preserving or improving effectiveness. Through multi-task, multi-stage training and flexible indexing workflows, ColBERTer achieves up to 2.5x storage reductions and can operate with as little as one dimension per token in Uni-ColBERTer, maintaining strong performance on MS MARCO and TREC-DL and showing robust out-of-domain results. The work also provides a transparent scoring interface mapping to whole words, facilitating interpretability and user-facing explanations in neural IR systems.

Abstract

Recent progress in neural information retrieval has demonstrated large gains in effectiveness, while often sacrificing the efficiency and interpretability of the neural model compared to classical approaches. This paper proposes ColBERTer, a neural retrieval model using contextualized late interaction (ColBERT) with enhanced reduction. Along the effectiveness Pareto frontier, ColBERTer's reductions dramatically lower ColBERT's storage requirements while simultaneously improving the interpretability of its token-matching scores. To this end, ColBERTer fuses single-vector retrieval, multi-vector refinement, and optional lexical matching components into one model. For its multi-vector component, ColBERTer reduces the number of stored vectors per document by learning unique whole-word representations for the terms in each document and learning to identify and remove word representations that are not essential to effective scoring. We employ an explicit multi-task, multi-stage training to facilitate using very small vector dimensions. Results on the MS MARCO and TREC-DL collection show that ColBERTer can reduce the storage footprint by up to 2.5x, while maintaining effectiveness. With just one dimension per token in its smallest setting, ColBERTer achieves index storage parity with the plaintext size, with very strong effectiveness results. Finally, we demonstrate ColBERTer's robustness on seven high-quality out-of-domain collections, yielding statistically significant gains over traditional retrieval baselines.
Paper Structure (29 sections, 20 equations, 5 figures, 5 tables)

This paper contains 29 sections, 20 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Tokenization
  4. BERT: Architecture
  5. ColBERT Architecture
  6. Related Work
  7. Vector Reduction
  8. Vector Compression
  9. .
  10. 2-Way Dimension Reduction
  11. :: Bag of Unique Whole-Words
  12. Simplified Contextualized Stopwords
  13. Matching & Score Aggregation
  14. Uni-ColBERTer: Extreme Reduction with Lexical Matching
  15. Model Lifecycle
  16. ...and 14 more sections

Figures (5)

  • Figure 1: Example of ColBERTer's BOW$^2$ (Bag Of Whole-Words): ColBERTer stores and matches unique whole-word representations. The words in BOW$^2$ are ordered by implicitly learned query-independent term importance. Matched words are highlighted in blue with whole-word scores displayed in a user-friendly way next to them.
  • Figure 2: The ColBERTer encoding architecture, followed by the query-time workflow. The passage representations (both the single CLS and token vectors) are pre-computed during indexing time. The enhanced reductions with the 2-way dimension reduction, the unique BOW$^2$ aggregation, contextualized stopwords and token dimensionality reduction are applied symmetrically to passages and queries (except for the stopword removal).
  • Figure 3: The potential retrieval and refine workflows of ColBERTer at query time. Broadly categorized by: full hybrid (➊), single index, then refine with the other (➋ + ➌), or only one index for ablation purposes (➍ + ➎).
  • Figure 4: Tradeoff between storage requirements and effectiveness on MSMARCO Dev. Note the log scale of the y-axis.
  • Figure 5: Effect size based evaluation of ColBERTer's zero-shot out of domain robustness. We compare three pairings between the control vs. the treatment retrieval method. The comparison is dependent on the effect size of each collection and the mean NDCG@10 differences are standardized with the effect size. The confidence intervals are plotted as interval around the standardized mean difference ◆. The Summary Effect of the treatment is computed with the Random-Effect (RE) model, here we see an overall significant improvement for ColBERTer (Dim1 and Dim32) to BM25.