Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Analysis on Matching Mechanisms and Token Pruning for Late-interaction Models

Qi Liu, Gang Guo, Jiaxin Mao, Zhicheng Dou, Ji-Rong Wen, Hao Jiang, Xinyu Zhang, Zhao Cao

TL;DR

This study systematically analyzes the matching mechanism of the recently proposed late-interaction multi-vector models and shows that the sum-of-max operation heavily relies on the co-occurrence signals and some important words in the document.

Abstract

With the development of pre-trained language models, the dense retrieval models have become promising alternatives to the traditional retrieval models that rely on exact match and sparse bag-of-words representations. Different from most dense retrieval models using a bi-encoder to encode each query or document into a dense vector, the recently proposed late-interaction multi-vector models (i.e., ColBERT and COIL) achieve state-of-the-art retrieval effectiveness by using all token embeddings to represent documents and queries and modeling their relevance with a sum-of-max operation. However, these fine-grained representations may cause unacceptable storage overhead for practical search systems. In this study, we systematically analyze the matching mechanism of these late-interaction models and show that the sum-of-max operation heavily relies on the co-occurrence signals and some important words in the document. Based on these findings, we then propose several simple document pruning methods to reduce the storage overhead and compare the effectiveness of different pruning methods on different late-interaction models. We also leverage query pruning methods to further reduce the retrieval latency. We conduct extensive experiments on both in-domain and out-domain datasets and show that some of the used pruning methods can significantly improve the efficiency of these late-interaction models without substantially hurting their retrieval effectiveness.

An Analysis on Matching Mechanisms and Token Pruning for Late-interaction Models

TL;DR

This study systematically analyzes the matching mechanism of the recently proposed late-interaction multi-vector models and shows that the sum-of-max operation heavily relies on the co-occurrence signals and some important words in the document.

Abstract

With the development of pre-trained language models, the dense retrieval models have become promising alternatives to the traditional retrieval models that rely on exact match and sparse bag-of-words representations. Different from most dense retrieval models using a bi-encoder to encode each query or document into a dense vector, the recently proposed late-interaction multi-vector models (i.e., ColBERT and COIL) achieve state-of-the-art retrieval effectiveness by using all token embeddings to represent documents and queries and modeling their relevance with a sum-of-max operation. However, these fine-grained representations may cause unacceptable storage overhead for practical search systems. In this study, we systematically analyze the matching mechanism of these late-interaction models and show that the sum-of-max operation heavily relies on the co-occurrence signals and some important words in the document. Based on these findings, we then propose several simple document pruning methods to reduce the storage overhead and compare the effectiveness of different pruning methods on different late-interaction models. We also leverage query pruning methods to further reduce the retrieval latency. We conduct extensive experiments on both in-domain and out-domain datasets and show that some of the used pruning methods can significantly improve the efficiency of these late-interaction models without substantially hurting their retrieval effectiveness.
Paper Structure (38 sections, 14 equations, 8 figures, 7 tables)

This paper contains 38 sections, 14 equations, 8 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional Retrieval Models
  4. Neural Retrieval Models
  5. Pruning for Inverted Index
  6. Analysis and Pruning for (Col)BERT
  7. Methodologies
  8. Preliminaries
  9. Analyses of late-interaction models
  10. Document Token Pruning
  11. First-$\alpha$ Pruning.
  12. IDF-Top-$\alpha$ Pruning.
  13. Attention-Top-$\alpha$ Pruning.
  14. End-to-End Retrieval and Query Token Pruning
  15. Experiment
  16. ...and 23 more sections

Figures (8)

  • Figure 1: Architectures of dense retrieval models.
  • Figure 2: The distribution of contributions of different tokens. Each bin consists of ten percent of the total tokens according to their relative position or IDF values. The y-axis shows the proportion contributing to the final scores of each bin.
  • Figure 3: The pipeline of Document Token Pruning
  • Figure 4: The pipeline of Query Token Pruning for ColBERT.
  • Figure 5: Effectiveness of different DTP methods of different models on MSMARCO Dev. The dashed line denotes the performance of models without DTP. The x-axis is the remaining ratio $\alpha$ and the y-axis of each sub-figure is the MRR@10 metric.
  • ...and 3 more figures