Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Measuring the Inconsistency of Large Language Models in Preferential Ranking

Xiutian Zhao, Ke Wang, Wei Peng

TL;DR

A formalization of consistency based on order theory is introduced, outlining criteria such as transitivity, asymmetry, reversibility, and independence from irrelevant alternatives and highlighting a significant inconsistency in LLM-generated preferential rankings.

Abstract

Despite large language models' (LLMs) recent advancements, their bias and hallucination issues persist, and their ability to offer consistent preferential rankings remains underexplored. This study investigates the capacity of LLMs to provide consistent ordinal preferences, a crucial aspect in scenarios with dense decision space or lacking absolute answers. We introduce a formalization of consistency based on order theory, outlining criteria such as transitivity, asymmetry, reversibility, and independence from irrelevant alternatives. Our diagnostic experiments on selected state-of-the-art LLMs reveal their inability to meet these criteria, indicating a strong positional bias and poor transitivity, with preferences easily swayed by irrelevant alternatives. These findings highlight a significant inconsistency in LLM-generated preferential rankings, underscoring the need for further research to address these limitations.

Measuring the Inconsistency of Large Language Models in Preferential Ranking

TL;DR

A formalization of consistency based on order theory is introduced, outlining criteria such as transitivity, asymmetry, reversibility, and independence from irrelevant alternatives and highlighting a significant inconsistency in LLM-generated preferential rankings.

Abstract

Despite large language models' (LLMs) recent advancements, their bias and hallucination issues persist, and their ability to offer consistent preferential rankings remains underexplored. This study investigates the capacity of LLMs to provide consistent ordinal preferences, a crucial aspect in scenarios with dense decision space or lacking absolute answers. We introduce a formalization of consistency based on order theory, outlining criteria such as transitivity, asymmetry, reversibility, and independence from irrelevant alternatives. Our diagnostic experiments on selected state-of-the-art LLMs reveal their inability to meet these criteria, indicating a strong positional bias and poor transitivity, with preferences easily swayed by irrelevant alternatives. These findings highlight a significant inconsistency in LLM-generated preferential rankings, underscoring the need for further research to address these limitations.

Paper Structure

This paper contains 15 sections, 2 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Experiment Setup and Preliminaries
  3. Definition
  4. Datasets
  5. Evaluated Models
  6. Preliminary Examinations
  7. Alternative Label Bias
  8. Question Format Sensitivity
  9. Main Experiments and Key Observations
  10. Irreflexivity
  11. Asymmetry and Transitivity
  12. Independent from Irrelevant Alternative
  13. Reversibility
  14. Conclusion and Future Work
  15. Reproducibility

Figures (2)

  • Figure 1: An example of violating Independence from Irrelevant Alternatives (IIA) criterion. Initially, given 3 choices, the model preferred Circle over Square over Triangle. However, after introducing a new choice Star, the relative preferential positions among the initial 3 choices inconsistently changed.
  • Figure 2: A 4-option binary comparison matrix (left) and a breakdown of its upper and lower triangles (right). Each triangular matrices can be transformed into a relation matrix for each relation.