Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Contextual Dual Learning Algorithm with Listwise Distillation for Unbiased Learning to Rank

Lulu Yu, Keping Bi, Shiyu Ni, Jiafeng Guo

TL;DR

A Contextual Dual Learning Algorithm with Listwise Distillation (CDLA-LD) to simultaneously address both position bias and contextual bias is proposed to simultaneously address both position bias and contextual bias.

Abstract

Unbiased Learning to Rank (ULTR) aims to leverage biased implicit user feedback (e.g., click) to optimize an unbiased ranking model. The effectiveness of the existing ULTR methods has primarily been validated on synthetic datasets. However, their performance on real-world click data remains unclear. Recently, Baidu released a large publicly available dataset of their web search logs. Subsequently, the NTCIR-17 ULTRE-2 task released a subset dataset extracted from it. We conduct experiments on commonly used or effective ULTR methods on this subset to determine whether they maintain their effectiveness. In this paper, we propose a Contextual Dual Learning Algorithm with Listwise Distillation (CDLA-LD) to simultaneously address both position bias and contextual bias. We utilize a listwise-input ranking model to obtain reconstructed feature vectors incorporating local contextual information and employ the Dual Learning Algorithm (DLA) method to jointly train this ranking model and a propensity model to address position bias. As this ranking model learns the interaction information within the documents list of the training set, to enhance the ranking model's generalization ability, we additionally train a pointwise-input ranking model to learn the listwise-input ranking model's capability for relevance judgment in a listwise manner. Extensive experiments and analysis confirm the effectiveness of our approach.

Contextual Dual Learning Algorithm with Listwise Distillation for Unbiased Learning to Rank

TL;DR

A Contextual Dual Learning Algorithm with Listwise Distillation (CDLA-LD) to simultaneously address both position bias and contextual bias is proposed to simultaneously address both position bias and contextual bias.

Abstract

Unbiased Learning to Rank (ULTR) aims to leverage biased implicit user feedback (e.g., click) to optimize an unbiased ranking model. The effectiveness of the existing ULTR methods has primarily been validated on synthetic datasets. However, their performance on real-world click data remains unclear. Recently, Baidu released a large publicly available dataset of their web search logs. Subsequently, the NTCIR-17 ULTRE-2 task released a subset dataset extracted from it. We conduct experiments on commonly used or effective ULTR methods on this subset to determine whether they maintain their effectiveness. In this paper, we propose a Contextual Dual Learning Algorithm with Listwise Distillation (CDLA-LD) to simultaneously address both position bias and contextual bias. We utilize a listwise-input ranking model to obtain reconstructed feature vectors incorporating local contextual information and employ the Dual Learning Algorithm (DLA) method to jointly train this ranking model and a propensity model to address position bias. As this ranking model learns the interaction information within the documents list of the training set, to enhance the ranking model's generalization ability, we additionally train a pointwise-input ranking model to learn the listwise-input ranking model's capability for relevance judgment in a listwise manner. Extensive experiments and analysis confirm the effectiveness of our approach.
Paper Structure (18 sections, 6 equations, 2 figures, 3 tables)

This paper contains 18 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Unbiased Learning to Rank (ULTR)
  4. ULTR on the Baidu-ULTR Dataset
  5. Preliminaries
  6. Problem Formulation
  7. Dual Learning Algorithm (DLA)
  8. Contextual Dual Learning Algorithm with Listwise Distillation (CDLA-LD)
  9. Overview
  10. Contextual Dual Learning Algorithm (CDLA)
  11. Listwise Distillation
  12. Experiments
  13. Experimental Settings
  14. Performance Comparison of Different ULTR Methods (RQ1)
  15. Propensity Analysis (RQ2)
  16. ...and 3 more sections

Figures (2)

  • Figure 1: The overall architecture of CDLA-LD, where we jointly train an unbiased listwise-input ranking model (a) employing Transformer Encoder and an unbiased propensity model (b). And simultaneously we learn a pointwise-input ranking model (c) distilled from (a) in a listwise manner.
  • Figure 2: (a) Normalized propensity estimated from ULTR methods that solely consider position factor to model the propensity model, compared to the mean CTR. (b) Normalized propensity estimated from CDLA-LD and C-DLA-LD.