Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Causal Information-Flow Framework for Unbiased Learning-to-Rank

Haoming Gong, Qingyao Ai, Zhihao Tao, Yongfeng Zhang

TL;DR

This work tackles biased user clicks in learning-to-rank by introducing a Structural Causal Model (SCM) that explicitly represents how bias propagates through exposure and post-exposure channels. It defines a causal information-flow framework that quantifies bias leakage via conditional mutual information along bias channels and regularizes learning with leakage penalties, supported by a doubly robust estimator to mitigate propensity misspecification. The authors prove a risk--divergence bound and a divergence--leakage bound, showing that the ranking risk gap is controlled by channel leakage and that leakage is identifiable under adjustment. Empirically, Structural Information-Flow (SIF) improves ranking performance and reduces leakage on Yahoo! LETOR and MSLR-WEB30K, especially when multiple biases (e.g., position and trust) interact, and provides practical budgets and diagnostics for debiasing in real systems.

Abstract

In web search and recommendation systems, user clicks are widely used to train ranking models. However, click data is heavily biased, i.e., users tend to click higher-ranked items (position bias), choose only what was shown to them (selection bias), and trust top results more (trust bias). Without explicitly modeling these biases, the true relevance of ranked items cannot be correctly learned from clicks. Existing Unbiased Learning-to-Rank (ULTR) methods mainly correct position bias and rely on propensity estimation, but they cannot measure remaining bias, provide risk guarantees, or jointly handle multiple bias sources. To overcome these challenges, this paper introduces a novel causal learning-based ranking framework that extends ULTR by combining Structural Causal Models (SCMs) with information-theoretic tools. SCMs specify how clicks are generated and help identify the true relevance signal from click data, while conditional mutual information, measures how much bias leaks into the learned relevance estimates. We use this leakage measure to define a rigorous notion of disentanglement and include it as a regularizer during model training to reduce bias. In addition, we incorporate a causal inference estimator, i.e., doubly robust estimator, to ensure more reliable risk estimation. Experiments on standard Learning-to-Rank benchmarks show that our method consistently reduces measured bias leakage and improves ranking performance, especially in realistic scenarios where multiple biases-such as position and trust bias-interact strongly.

A Causal Information-Flow Framework for Unbiased Learning-to-Rank

TL;DR

This work tackles biased user clicks in learning-to-rank by introducing a Structural Causal Model (SCM) that explicitly represents how bias propagates through exposure and post-exposure channels. It defines a causal information-flow framework that quantifies bias leakage via conditional mutual information along bias channels and regularizes learning with leakage penalties, supported by a doubly robust estimator to mitigate propensity misspecification. The authors prove a risk--divergence bound and a divergence--leakage bound, showing that the ranking risk gap is controlled by channel leakage and that leakage is identifiable under adjustment. Empirically, Structural Information-Flow (SIF) improves ranking performance and reduces leakage on Yahoo! LETOR and MSLR-WEB30K, especially when multiple biases (e.g., position and trust) interact, and provides practical budgets and diagnostics for debiasing in real systems.

Abstract

In web search and recommendation systems, user clicks are widely used to train ranking models. However, click data is heavily biased, i.e., users tend to click higher-ranked items (position bias), choose only what was shown to them (selection bias), and trust top results more (trust bias). Without explicitly modeling these biases, the true relevance of ranked items cannot be correctly learned from clicks. Existing Unbiased Learning-to-Rank (ULTR) methods mainly correct position bias and rely on propensity estimation, but they cannot measure remaining bias, provide risk guarantees, or jointly handle multiple bias sources. To overcome these challenges, this paper introduces a novel causal learning-based ranking framework that extends ULTR by combining Structural Causal Models (SCMs) with information-theoretic tools. SCMs specify how clicks are generated and help identify the true relevance signal from click data, while conditional mutual information, measures how much bias leaks into the learned relevance estimates. We use this leakage measure to define a rigorous notion of disentanglement and include it as a regularizer during model training to reduce bias. In addition, we incorporate a causal inference estimator, i.e., doubly robust estimator, to ensure more reliable risk estimation. Experiments on standard Learning-to-Rank benchmarks show that our method consistently reduces measured bias leakage and improves ranking performance, especially in realistic scenarios where multiple biases-such as position and trust bias-interact strongly.
Paper Structure (44 sections, 2 theorems, 55 equations, 5 figures, 3 tables)

This paper contains 44 sections, 2 theorems, 55 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Preliminaries
  4. Logging vs. Idealized Settings (obs vs. tgt)
  5. Radon–Nikodym / IPS Derivation for Binary Exposure
  6. Limitations of IPS and Motivation for Our Framework
  7. Misspecified exposure probabilities.
  8. Hidden factors affecting both showing and relevance.
  9. Additional bias channels beyond exposure.
  10. Our Proposed Causal Information-Flow Framework
  11. SCM with Bias Channels and Information Flow
  12. Bias Sources, Bias Channels, and Disentanglement
  13. Disentanglement.
  14. Channel divergences.
  15. Risk--Divergence Bound: From Risk Gap to Channel Mechanisms
  16. ...and 29 more sections

Key Result

Theorem 1

Assume $0\le L(c;\theta)\le L_{\max}$ and that, conditioned on $(X,K)$, the click distributions under $p_{\mathrm{obs}}$ and $p_{\mathrm{tgt}}$ differ only through the bias-channel vector $Y=(Y_1,\dots,Y_m)$. Then: Moreover, if the channels $Y_j$ are conditionally independent given $(X,K)$, the joint divergence decomposes into the sum of channel divergences:

Figures (5)

  • Figure 1: SCM for ranking and clicks with exposure channel $E$ (position and presentation effects) and an optional downstream decision/trust node $D$. Any influence of upstream quantities on the click $C$ must pass through exposure and relevance. Our framework uses this step-by-step structure: only information that actually travels through these channels can distort learning.
  • Figure 2: Leakage (in nats, where 1 nat $\approx$ 1.44 bits) and effective sample size (ESS) over epochs on Yahoo! LETOR. SIF reduces leakage to the 0.02-nat target budget while substantially improving ESS relative to DLA. Higher ESS indicates more efficient use of training data with lower variance in the debiasing estimator---SIF's ESS grows from 1900 to 2520 (+33%), while DLA's grows only from 1880 to 2100 (+12%).
  • Figure 3: Exposure propensity mean absolute error (MAE) to the PBM oracle on the position+trust Yahoo! simulator (lower is better). Bars correspond to DLA (baseline), SIF-full (A4, MI on both exposure and trust channels), SIF-DR (A4+DR), SIF-exp (MI on exposure only), and SIF-dec (MI on trust only). SIF-full achieves the lowest MAE (best propensity alignment), while SIF-DR trades slightly higher MAE for improved downstream NDCG/MRR.
  • Figure 4: Channel-wise MI ablation under position+trust bias on Yahoo! LETOR. Regularising either exposure (Exp-MI) or trust (Dec-MI) improves NDCG over DLA; trust-channel MI performs best, while SIF+DR (SIF-DR) offers a strong and robust compromise.
  • Figure 5: MI trajectories on the position+trust Yahoo! simulator. SIF variants steadily contract total leakage (measured in nats), whereas DLA remains at a higher, relatively flat level.

Theorems & Definitions (6)

  • Theorem 1: Risk--divergence bound
  • Theorem 2: Divergence--leakage bound
  • proof
  • proof
  • proof
  • proof