Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Passage Embeddings for Efficient Listwise Reranking with Large Language Models

Qi Liu, Bo Wang, Nan Wang, Jiaxin Mao

TL;DR

This paper tackles the inefficiency of listwise reranking with large language models by introducing PE-Rank, which compresses multiple passages into a compact context using passage embeddings treated as special tokens. A two-stage training regime aligns the embedding space and trains for ranking, while a Dynamic-Constrained Decoding strategy confines the LLM's output to a small set of passage tokens, drastically reducing input length and decoding cost. Empirical results on TREC DL and BEIR show PE-Rank achieves competitive ranking performance with substantial efficiency gains, including multi-fold reductions in latency and a practical 0.5-second per-query time for top-20 reranking. The approach demonstrates that embedding-based context compression can make online LLM-based listwise reranking feasible, with code available for reproduction.

Abstract

Recent studies have demonstrated the effectiveness of using large language language models (LLMs) in passage ranking. The listwise approaches, such as RankGPT, have become new state-of-the-art in this task. However, the efficiency of RankGPT models is limited by the maximum context length and relatively high latency of LLM inference. To address these issues, in this paper, we propose PE-Rank, leveraging the single passage embedding as a good context compression for efficient listwise passage reranking. By treating each passage as a special token, we can directly input passage embeddings into LLMs, thereby reducing input length. Additionally, we introduce an inference method that dynamically constrains the decoding space to these special tokens, accelerating the decoding process. For adapting the model to reranking, we employ listwise learning to rank loss for training. Evaluation results on multiple benchmarks demonstrate that PE-Rank significantly improves efficiency in both prefilling and decoding, while maintaining competitive ranking effectiveness. The Code is available at https://github.com/liuqi6777/pe_rank.

Leveraging Passage Embeddings for Efficient Listwise Reranking with Large Language Models

TL;DR

This paper tackles the inefficiency of listwise reranking with large language models by introducing PE-Rank, which compresses multiple passages into a compact context using passage embeddings treated as special tokens. A two-stage training regime aligns the embedding space and trains for ranking, while a Dynamic-Constrained Decoding strategy confines the LLM's output to a small set of passage tokens, drastically reducing input length and decoding cost. Empirical results on TREC DL and BEIR show PE-Rank achieves competitive ranking performance with substantial efficiency gains, including multi-fold reductions in latency and a practical 0.5-second per-query time for top-20 reranking. The approach demonstrates that embedding-based context compression can make online LLM-based listwise reranking feasible, with code available for reproduction.

Abstract

Recent studies have demonstrated the effectiveness of using large language language models (LLMs) in passage ranking. The listwise approaches, such as RankGPT, have become new state-of-the-art in this task. However, the efficiency of RankGPT models is limited by the maximum context length and relatively high latency of LLM inference. To address these issues, in this paper, we propose PE-Rank, leveraging the single passage embedding as a good context compression for efficient listwise passage reranking. By treating each passage as a special token, we can directly input passage embeddings into LLMs, thereby reducing input length. Additionally, we introduce an inference method that dynamically constrains the decoding space to these special tokens, accelerating the decoding process. For adapting the model to reranking, we employ listwise learning to rank loss for training. Evaluation results on multiple benchmarks demonstrate that PE-Rank significantly improves efficiency in both prefilling and decoding, while maintaining competitive ranking effectiveness. The Code is available at https://github.com/liuqi6777/pe_rank.
Paper Structure (31 sections, 8 equations, 4 figures, 10 tables, 1 algorithm)

This paper contains 31 sections, 8 equations, 4 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models as Rerankers
  4. Context Compression
  5. Methodology
  6. Overview
  7. Inference
  8. Training
  9. Alignment stage
  10. Learning-to-rank stage
  11. Experiment Setup
  12. Model selection
  13. Training Data
  14. Evaluation Datasets
  15. Baselines
  16. ...and 16 more sections

Figures (4)

  • Figure 1: Comparison between RankGPT (upper) and PE-Rank (lower). RankGPT takes the whole passage as input and outputs ordered numbers, while PE-Rank takes a list of special tokens as both input and output. On the right side, we show the reranking results on DL19 using different forms of inputs.
  • Figure 2: Overview of PE-Rank under a two-stage ranking paradigm. (a) is retrieval stage, retrieve $n$ passage embeddings; (b) is the forward pass procedure of LLM; (c) shows the listwise decoding process.
  • Figure 3: Illustration of two types of training data and the learning-to-rank training process.
  • Figure 4: Latency of reranking top 100 candidates at different stages during inference. $p$ (passage), $s$ (summary), and $t$ (title) denote the different forms of inputs of RankMistral.