Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Ranking

Yifan Zeng, Ojas Tendolkar, Raymond Baartmans, Qingyun Wu, Lizhong Chen, Huazheng Wang

TL;DR

This paper identifies two kinds of intrinsic inconsistency in LLM-based pairwise comparisons: order inconsistency which leads to conflicting results when switching the passage order, and transitive inconsistency which leads to non-transitive triads among all preference pairs.

Abstract

Ranking passages by prompting a large language model (LLM) can achieve promising performance in modern information retrieval (IR) systems. A common approach to sort the ranking list is by prompting LLMs for a pairwise or setwise comparison which often relies on sorting algorithms. However, sorting-based methods require consistent comparisons to correctly sort the passages, which we show that LLMs often violate. We identify two kinds of intrinsic inconsistency in LLM-based pairwise comparisons: order inconsistency which leads to conflicting results when switching the passage order, and transitive inconsistency which leads to non-transitive triads among all preference pairs. Our study of these inconsistencies is relevant for understanding and improving the stability of any ranking scheme based on relative preferences. In this paper, we propose LLM-RankFusion, an LLM-based ranking framework that mitigates these inconsistencies and produces a robust ranking list. LLM-RankFusion mitigates order inconsistency using in-context learning (ICL) to demonstrate order-agnostic comparisons and calibration to estimate the underlying preference probability between two passages. We then address transitive inconsistency by aggregating the ranking results from multiple rankers. In our experiments, we empirically show that LLM-RankFusion can significantly reduce inconsistent comparison results, improving the ranking quality by making the final ranking list more robust. Our code is available at \href{https://github.com/XHMY/LLM-RankFusion}{https://github.com/XHMY/LLM-RankFusion}

LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Ranking

TL;DR

This paper identifies two kinds of intrinsic inconsistency in LLM-based pairwise comparisons: order inconsistency which leads to conflicting results when switching the passage order, and transitive inconsistency which leads to non-transitive triads among all preference pairs.

Abstract

Ranking passages by prompting a large language model (LLM) can achieve promising performance in modern information retrieval (IR) systems. A common approach to sort the ranking list is by prompting LLMs for a pairwise or setwise comparison which often relies on sorting algorithms. However, sorting-based methods require consistent comparisons to correctly sort the passages, which we show that LLMs often violate. We identify two kinds of intrinsic inconsistency in LLM-based pairwise comparisons: order inconsistency which leads to conflicting results when switching the passage order, and transitive inconsistency which leads to non-transitive triads among all preference pairs. Our study of these inconsistencies is relevant for understanding and improving the stability of any ranking scheme based on relative preferences. In this paper, we propose LLM-RankFusion, an LLM-based ranking framework that mitigates these inconsistencies and produces a robust ranking list. LLM-RankFusion mitigates order inconsistency using in-context learning (ICL) to demonstrate order-agnostic comparisons and calibration to estimate the underlying preference probability between two passages. We then address transitive inconsistency by aggregating the ranking results from multiple rankers. In our experiments, we empirically show that LLM-RankFusion can significantly reduce inconsistent comparison results, improving the ranking quality by making the final ranking list more robust. Our code is available at \href{https://github.com/XHMY/LLM-RankFusion}{https://github.com/XHMY/LLM-RankFusion}
Paper Structure (29 sections, 5 equations, 1 figure, 15 tables, 1 algorithm)

This paper contains 29 sections, 5 equations, 1 figure, 15 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Inconsistency of LLM-based ranking
  4. Inconsistency of Pairwise Comparisons
  5. Inconsistency Measurement
  6. The Impact of Inconsistency
  7. Addressing LLM-based Ranking Inconsistency
  8. Mitigating Order Inconsistency
  9. In-Context Learning (ICL)
  10. Calibration
  11. LLM Ranking Aggregation
  12. Aggregation Across Sorting Methods
  13. Aggregation Across LLMs
  14. Experiments
  15. Experimental Setup
  16. ...and 14 more sections

Figures (1)

  • Figure 1: The LLM-RankFusion pipeline. It shows an illustration of the aggregation process to mitigate the transitive inconsistency. The ranking list proposals are formed by different rankers, and the details of each ranker are shown in the lower left of the figure. Each ranker includes ICL and calibration to address order inconsistency.