Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers

TL;DR

This paper tackles the inefficiency of generation-based zero-shot re-ranking with LLMs by introducing in-context re-ranking (ICR), a non-generative method that uses attention distributions to rank documents. By aggregating query–document attention across tokens, layers, and heads and calibrating with a content-free query, ICR yields calibrated token-level scores that sum to robust document rankings using only two forward passes, i.e., $O(1)$ complexity with respect to the document count. Empirical results on open-weight LLMs (Mistral 7B and Llama-3.1 8B) show ICR outperforms RankGPT on most single-hop and multi-hop benchmarks and substantially reduces latency (over 60%), sometimes approaching or matching proprietary models like GPT-4o mini. The study highlights that LLM attention carries rich re-ranking signals (contextualization, contradiction-based reasoning, and bridge-entity information) that can be effectively exploited without generation, suggesting broader potential for open-weight models beyond text generation. Overall, ICR demonstrates a practical, scalable direction for leveraging large language models in IR and motivates further exploration of attention-based strategies for other tasks.

Abstract

Information retrieval (IR) systems have played a vital role in modern digital life and have cemented their continued usefulness in this new era of generative AI via retrieval-augmented generation. With strong language processing capabilities and remarkable versatility, large language models (LLMs) have become popular choices for zero-shot re-ranking in IR systems. So far, LLM-based re-ranking methods rely on strong generative capabilities, which restricts their use to either specialized or powerful proprietary models. Given these restrictions, we ask: is autoregressive generation necessary and optimal for LLMs to perform re-ranking? We hypothesize that there are abundant signals relevant to re-ranking within LLMs that might not be used to their full potential via generation. To more directly leverage such signals, we propose in-context re-ranking (ICR), a novel method that leverages the change in attention pattern caused by the search query for accurate and efficient re-ranking. To mitigate the intrinsic biases in LLMs, we propose a calibration method using a content-free query. Due to the absence of generation, ICR only requires two ($O(1)$) forward passes to re-rank $N$ documents, making it substantially more efficient than generative re-ranking methods that require at least $O(N)$ forward passes. Our novel design also enables ICR to be applied to any LLM without specialized training while guaranteeing a well-formed ranking. Extensive experiments with two popular open-weight LLMs on standard single-hop and multi-hop information retrieval benchmarks show that ICR outperforms RankGPT while cutting the latency by more than 60% in practice. Through detailed analyses, we show that ICR's performance is specially strong on tasks that require more complex re-ranking signals. Our findings call for further exploration on novel ways of utilizing open-weight LLMs beyond text generation.

Figures (5)

• Figure 1: (a) Overview of in-context re-ranking (ICR). (b) Re-ranking performance using Llama-3.1 8B on single-hop and multi-hop re-ranking tasks. We report the micro-average across tasks.
• Figure 2: Illustration for In-context Re-ranking (ICR). ICR first aggregates (Agg) attention weights between document and query tokens to form token-level query ranking scores. Then a calibration score is calculated similarly with the calibration query, which is subtracted form the query ranking score. The final score is the sum of calibrated scores for all tokens for each document.
• Figure 3: Scaling trend of performance and average latency on the SciFact dataset.
• Figure 4: Average calibration scores of ICR at different document input positions.
• Figure G.1: Calibration Score of Mistral 7B.