Identifiability Matters: Revealing the Hidden Recoverable Condition in Unbiased Learning to Rank

TL;DR

This work reveals that recovering true relevance from biased click logs in ULTR is not guaranteed and formalizes identifiability as recovering $r(x)$ up to a scalar when $r(x)o(t)=r'(x)o'(t)$. It shows identifiability is equivalent to the connectivity of an identifiability graph (IG) constructed from the dataset, and introduces two data-editing methods—node intervention (online swaps guided by an MST) and node merging (offline merging of bias nodes with bounded error)—to restore IG connectivity. Empirical validation on fully synthetic data, Yahoo! and Istella-S, and TianGong-ST demonstrates that identifiability improves ranking performance, and that excessive bias factors can induce unidentifiability in real data. The results offer practical, model-agnostic remedies for ULTR bias and highlight the importance of dataset design and initialization in achieving reliable relevance recovery.

Abstract

Unbiased Learning to Rank (ULTR) aims to train unbiased ranking models from biased click logs, by explicitly modeling a generation process for user behavior and fitting click data based on examination hypothesis. Previous research found empirically that the true latent relevance is mostly recoverable through click fitting. However, we demonstrate that this is not always achievable, resulting in a significant reduction in ranking performance. This research investigates the conditions under which relevance can be recovered from click data in the first principle. We initially characterize a ranking model as identifiable if it can recover the true relevance up to a scaling transformation, a criterion sufficient for the pairwise ranking objective. Subsequently, we investigate an equivalent condition for identifiability, articulated as a graph connectivity test problem: the recovery of relevance is feasible if and only if the identifiability graph (IG), derived from the underlying structure of the dataset, is connected. The presence of a disconnected IG may lead to degenerate cases and suboptimal ranking performance. To tackle this challenge, we introduce two methods, namely node intervention and node merging, designed to modify the dataset and restore the connectivity of the IG. Empirical results derived from a simulated dataset and two real-world LTR benchmark datasets not only validate our proposed theory but also demonstrate the effectiveness of our methods in alleviating data bias when the relevance model is unidentifiable.

Figures (7)

• Figure 1: Examples for identifiable case and unidentifiable case.
• Figure 2: Illustrations for the proposed two methods to deal with unidentifiable datasets. In node intervention, we swap Feature B related to Bias 3 to Bias 2, which connects the two disconnected nodes. In node merging, we merge Bias 2 and Bias 3 into a new node 2 & 3, which indicates that Bias 2 and Bias 3 will have the same estimated observation. Both methods are applied to datasets before ULTR training.
• Figure 3: (a) Performance of DLA on different numbers of connected components $K$ across different click counts. Shadowed areas depict variance. (b) Performance of different methods across different $K$. NI = Node Intervention. NM = Node Merging.
• Figure 4: (a) Impact of cost selection strategies in node intervention. (b) Impact of merging strategies in node merging. (c)(d) Impact of the number of context types on the identifiability probabilities on the two datasets.
• Figure 5: IGs of the simulated datasets. Numbers in the nodes denote the position index (starting from 0 to 9).

Theorems & Definitions (17)

• Example 1
• Definition 1: Identifiability
• Theorem 1: Main result: Equivalent condition of identifiability
• Remark 1
• Example 2: Estimation of identifiability probability
• Remark 2
• Proposition 1
• Remark 3
• Proposition 2: Error bound of merging two components
• Remark 4