Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Set-Encoder: Permutation-Invariant Inter-Passage Attention for Listwise Passage Re-Ranking with Cross-Encoders

Ferdinand Schlatt, Maik Fröbe, Harrisen Scells, Shengyao Zhuang, Bevan Koopman, Guido Zuccon, Benno Stein, Martin Potthast, Matthias Hagen

TL;DR

The paper tackles permutation sensitivity in cross-encoder passage re-ranking by introducing the Set-Encoder, a permutation-invariant architecture with inter-passage attention implemented via dedicated [INT] interaction tokens. Each passage is processed in parallel, and the [INT] tokens enable lightweight cross-passage information exchange while keeping positional encodings independent of passage order, leading to robust and efficient ranking. Empirically, the Set-Encoder matches state-of-the-art listwise models on Cranfield-style and TIREx datasets and often outperforms them in permutation-perturbed settings, while being substantially more efficient. When fine-tuned for novelty and duplicates, the model shows improved novelty-aware rankings, illustrating that permutation-invariant interactions can be leveraged to boost performance in specific ranking objectives; overall, permutation invariance emerges as a key factor for both effectiveness and efficiency in re-ranking.

Abstract

Existing cross-encoder models can be categorized as pointwise, pairwise, or listwise. Pairwise and listwise models allow passage interactions, which typically makes them more effective than pointwise models but less efficient and less robust to input passage order permutations. To enable efficient permutation-invariant passage interactions during re-ranking, we propose a new cross-encoder architecture with inter-passage attention: the Set-Encoder. In experiments on TREC Deep Learning and TIREx, the Set-Encoder is as effective as state-of-the-art listwise models while being more efficient and invariant to input passage order permutations. Compared to pointwise models, the Set-Encoder is particularly more effective when considering inter-passage information, such as novelty, and retains its advantageous properties compared to other listwise models. Our code is publicly available at https://github.com/webis-de/ECIR-25.

Set-Encoder: Permutation-Invariant Inter-Passage Attention for Listwise Passage Re-Ranking with Cross-Encoders

TL;DR

The paper tackles permutation sensitivity in cross-encoder passage re-ranking by introducing the Set-Encoder, a permutation-invariant architecture with inter-passage attention implemented via dedicated [INT] interaction tokens. Each passage is processed in parallel, and the [INT] tokens enable lightweight cross-passage information exchange while keeping positional encodings independent of passage order, leading to robust and efficient ranking. Empirically, the Set-Encoder matches state-of-the-art listwise models on Cranfield-style and TIREx datasets and often outperforms them in permutation-perturbed settings, while being substantially more efficient. When fine-tuned for novelty and duplicates, the model shows improved novelty-aware rankings, illustrating that permutation-invariant interactions can be leveraged to boost performance in specific ranking objectives; overall, permutation invariance emerges as a key factor for both effectiveness and efficiency in re-ranking.

Abstract

Existing cross-encoder models can be categorized as pointwise, pairwise, or listwise. Pairwise and listwise models allow passage interactions, which typically makes them more effective than pointwise models but less efficient and less robust to input passage order permutations. To enable efficient permutation-invariant passage interactions during re-ranking, we propose a new cross-encoder architecture with inter-passage attention: the Set-Encoder. In experiments on TREC Deep Learning and TIREx, the Set-Encoder is as effective as state-of-the-art listwise models while being more efficient and invariant to input passage order permutations. Compared to pointwise models, the Set-Encoder is particularly more effective when considering inter-passage information, such as novelty, and retains its advantageous properties compared to other listwise models. Our code is publicly available at https://github.com/webis-de/ECIR-25.
Paper Structure (22 sections, 5 equations, 3 figures, 4 tables)

This paper contains 22 sections, 5 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Set-Encoder Model
  4. Permutation-Invariant Input Encoding
  5. Inter-Passage Attention
  6. Fine-tuning
  7. Fine-tuning to Detect Duplicates
  8. Fine-tuning to Rank According to Novelty
  9. Evaluation
  10. Experimental Setup
  11. Effectiveness on Cranfield-Style Datasets
  12. In-domain Re-ranking.
  13. Out-of-domain Re-ranking.
  14. Passage Interaction "Unneccessity".
  15. Permutation Invariance.
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Comparison of our Set-Encoder architecture (top) with state-of-the-art listwise re-rankers (bottom) for three input passages. List-wise re-rankers concatenate the input passages, leading to potentially inconsistent rankings for different concatenation orderings. Many (or all) permutations are thus re-ranked for optimization. Instead, the Set-Encoder uses novel [INT] tokens for permutation-invariant inter-passage attention.
  • Figure 2: Proportional rank changes of various cross-encoders for re-ranking BM25 averaged across all queries from the TREC Deep Learning 2019 and 2020 tracks.
  • Figure 3: Effectiveness (nDCG@10) on different permutations of BM25 rankings for TREC Deep Learning 2019 and 2020.