Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RankSHAP: Shapley Value Based Feature Attributions for Learning to Rank

Tanya Chowdhury, Yair Zick, James Allan

TL;DR

RankSHAP introduces an axiomatic, Shapley-value-based framework for learning-to-rank explanations, addressing inconsistencies observed in prior ranking attributions. By redefining ranking value functions with Relevance Sensitivity and Position Sensitivity and proving a Generalized Ranking Effectiveness Metric (GREM), the authors show that RankSHAP is the unique attribution method that satisfies Rank-Efficiency, Rank-Missingness, Rank-Symmetry, and Rank-Monotonicity. They provide a practical KernelSHAP-based approximation, demonstrate superior fidelity over baselines on MS MARCO and Robust04 using multiple rankers, and corroborate human-aligned intuition via a user study. The work also offers an axiomatic comparison of existing ranking attribution methods, highlighting RankSHAP’s additive, decomposable value function as essential for reliable explanations in IR. Overall, RankSHAP promises axiomatically grounded, generalizable, and interpretable explanations for IR ranking models, with planned release of code.

Abstract

Numerous works propose post-hoc, model-agnostic explanations for learning to rank, focusing on ordering entities by their relevance to a query through feature attribution methods. However, these attributions often weakly correlate or contradict each other, confusing end users. We adopt an axiomatic game-theoretic approach, popular in the feature attribution community, to identify a set of fundamental axioms that every ranking-based feature attribution method should satisfy. We then introduce Rank-SHAP, extending classical Shapley values to ranking. We evaluate the RankSHAP framework through extensive experiments on two datasets, multiple ranking methods and evaluation metrics. Additionally, a user study confirms RankSHAP's alignment with human intuition. We also perform an axiomatic analysis of existing rank attribution algorithms to determine their compliance with our proposed axioms. Ultimately, our aim is to equip practitioners with a set of axiomatically backed feature attribution methods for studying IR ranking models, that ensure generality as well as consistency.

RankSHAP: Shapley Value Based Feature Attributions for Learning to Rank

TL;DR

RankSHAP introduces an axiomatic, Shapley-value-based framework for learning-to-rank explanations, addressing inconsistencies observed in prior ranking attributions. By redefining ranking value functions with Relevance Sensitivity and Position Sensitivity and proving a Generalized Ranking Effectiveness Metric (GREM), the authors show that RankSHAP is the unique attribution method that satisfies Rank-Efficiency, Rank-Missingness, Rank-Symmetry, and Rank-Monotonicity. They provide a practical KernelSHAP-based approximation, demonstrate superior fidelity over baselines on MS MARCO and Robust04 using multiple rankers, and corroborate human-aligned intuition via a user study. The work also offers an axiomatic comparison of existing ranking attribution methods, highlighting RankSHAP’s additive, decomposable value function as essential for reliable explanations in IR. Overall, RankSHAP promises axiomatically grounded, generalizable, and interpretable explanations for IR ranking models, with planned release of code.

Abstract

Numerous works propose post-hoc, model-agnostic explanations for learning to rank, focusing on ordering entities by their relevance to a query through feature attribution methods. However, these attributions often weakly correlate or contradict each other, confusing end users. We adopt an axiomatic game-theoretic approach, popular in the feature attribution community, to identify a set of fundamental axioms that every ranking-based feature attribution method should satisfy. We then introduce Rank-SHAP, extending classical Shapley values to ranking. We evaluate the RankSHAP framework through extensive experiments on two datasets, multiple ranking methods and evaluation metrics. Additionally, a user study confirms RankSHAP's alignment with human intuition. We also perform an axiomatic analysis of existing rank attribution algorithms to determine their compliance with our proposed axioms. Ultimately, our aim is to equip practitioners with a set of axiomatically backed feature attribution methods for studying IR ranking models, that ensure generality as well as consistency.
Paper Structure (50 sections, 3 theorems, 30 equations, 3 figures, 5 tables)

This paper contains 50 sections, 3 theorems, 30 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Our Contributions.
  3. Background
  4. Basic Axioms and the Shapley Value
  5. Document Relevance and IR Evaluation Metrics
  6. The RankSHAP Framework
  7. The Ranking attribution Problem
  8. Shapley Axioms for Ranking
  9. A Generalized Ranking Effectiveness Metric ($\textit{GREM}\xspace$)
  10. RankSHAP Attributions
  11. Approximating RankSHAP
  12. Axiomatic Analysis of Rank Feature Attribution algorithms
  13. Optimization Function
  14. Computational and Human Evaluation Studies
  15. Experimental Settings
  16. ...and 35 more sections

Key Result

Theorem 3.1

An ordered list evaluation metric satisfies the axioms of Relevance Sensitivity and Position Sensitivity if and only if it can be represented as where $g(rel_j)$ is a non-decreasing function of the relevance score $rel_j$ (gain function), and $h(j)$ is a non-negative, non-increasing function of the rank position $j$ (discount function).

Figures (3)

  • Figure 1: A user study instance, depicting 5 passages and a corresponding feature attribution. Words that influenced the ranking decision positively are shown in green, whereas those that influenced negatively were shown in red. Participants were asked to re-rank documents based on the feature attribution to try and estimate the query.
  • Figure 2: User Study Passage Snapshot
  • Figure 3: User Study Bar Chart and questions snapshot

Theorems & Definitions (5)

  • Theorem 3.1
  • Theorem 3.2
  • proof
  • Theorem A.1
  • proof