Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating the Effects of Sparse Attention on Cross-Encoders

Ferdinand Schlatt, Maik Fröbe, Matthias Hagen

TL;DR

This work systematically analyzes how token interactions can be reduced without harming the re-ranking effectiveness and finds that the query tokens do not need to attend to the passage or document tokens for effective re-ranking and that very small window sizes suffice.

Abstract

Cross-encoders are effective passage and document re-rankers but less efficient than other neural or classic retrieval models. A few previous studies have applied windowed self-attention to make cross-encoders more efficient. However, these studies did not investigate the potential and limits of different attention patterns or window sizes. We close this gap and systematically analyze how token interactions can be reduced without harming the re-ranking effectiveness. Experimenting with asymmetric attention and different window sizes, we find that the query tokens do not need to attend to the passage or document tokens for effective re-ranking and that very small window sizes suffice. In our experiments, even windows of 4 tokens still yield effectiveness on par with previous cross-encoders while reducing the memory requirements by at least 22% / 59% and being 1% / 43% faster at inference time for passages / documents.

Investigating the Effects of Sparse Attention on Cross-Encoders

TL;DR

This work systematically analyzes how token interactions can be reduced without harming the re-ranking effectiveness and finds that the query tokens do not need to attend to the passage or document tokens for effective re-ranking and that very small window sizes suffice.

Abstract

Cross-encoders are effective passage and document re-rankers but less efficient than other neural or classic retrieval models. A few previous studies have applied windowed self-attention to make cross-encoders more efficient. However, these studies did not investigate the potential and limits of different attention patterns or window sizes. We close this gap and systematically analyze how token interactions can be reduced without harming the re-ranking effectiveness. Experimenting with asymmetric attention and different window sizes, we find that the query tokens do not need to attend to the passage or document tokens for effective re-ranking and that very small window sizes suffice. In our experiments, even windows of 4 tokens still yield effectiveness on par with previous cross-encoders while reducing the memory requirements by at least 22% / 59% and being 1% / 43% faster at inference time for passages / documents.
Paper Structure (16 sections, 4 equations, 2 figures, 3 tables)

This paper contains 16 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sparse Asymmetric Attention Using Cross-Encoders
  4. Preliminaries
  5. Windowed Self-Attention
  6. Cross-Attention
  7. Locally Windowed Cross-Attention
  8. Experimental Setup
  9. Empirical Evaluation
  10. Effectiveness Results
  11. In-domain Effectiveness
  12. Out-of-domain Effectiveness
  13. Efficiency Results
  14. Windowed Matrix Multiplication
  15. Cross-Encoders Models
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Previous cross-encoder attention patterns (a, b, and c) and our newly proposed sparse cross-encoder (d). The marginal boxes denote input tokens (black: [CLS], blue: query, yellow: passage / document, red: start of sentence). The inner green boxes indicate token attention. Our new pattern considers the sub-sequences separately (indicated by the added spacing) and is asymmetric.
  • Figure 2: Comparison of windowed matrix multiplication kernels (a) and sparse cross-encoder models (b) in terms of efficiency. Time (ms / Document) and space (GB) efficiency are reported for window sizes $w \in \{4, 64\}$. All plots use a logarithmic scale with base $2$ on the x-axis and base $10$ on the y-axis.