Unlearning for Federated Online Learning to Rank: A Reproducibility Study

TL;DR

The paper addresses unlearning in Federated Online Learning to Rank (FOLTR) by adapting five unlearning strategies (Retraining, Fine-tuning, FedEraser, FedRemove, Gradient Ascent) to the federated ranking setting and evaluating them with a unified pipeline. It systematically analyzes under-unlearning and over-unlearning using data- and model-poisoning attacks, RelR, and nDCG@10 across four OLTR datasets, plus distribution-based Fidelity metrics. Key findings show that Fine-tuning provides the strongest under-unlearning performance and fastest recovery, while FedRemove often underperforms; Gradient Ascent can lead to over-unlearning, and FedEraser offers competitive results with calibration. The work advances reproducibility in FU for FOLTR and offers practical insights for privacy-preserving online ranking systems, accompanied by an open-source codebase to support future research.

Abstract

This paper reports on findings from a comparative study on the effectiveness and efficiency of federated unlearning strategies within Federated Online Learning to Rank (FOLTR), with specific attention to systematically analysing the unlearning capabilities of methods in a verifiable manner. Federated approaches to ranking of search results have recently garnered attention to address users privacy concerns. In FOLTR, privacy is safeguarded by collaboratively training ranking models across decentralized data sources, preserving individual user data while optimizing search results based on implicit feedback, such as clicks. Recent legislation introduced across numerous countries is establishing the so called "the right to be forgotten", according to which services based on machine learning models like those in FOLTR should provide capabilities that allow users to remove their own data from those used to train models. This has sparked the development of unlearning methods, along with evaluation practices to measure whether unlearning of a user data successfully occurred. Current evaluation practices are however often controversial, necessitating the use of multiple metrics for a more comprehensive assessment -- but previous proposals of unlearning methods only used single evaluation metrics. This paper addresses this limitation: our study rigorously assesses the effectiveness of unlearning strategies in managing both under-unlearning and over-unlearning scenarios using adapted, and newly proposed evaluation metrics. Thanks to our detailed analysis, we uncover the strengths and limitations of five unlearning strategies, offering valuable insights into optimizing federated unlearning to balance data privacy and system performance within FOLTR. We publicly release our code and complete results at https://github.com/Iris1026/Unlearning-for-FOLTR.git.

Figures (3)

• Figure 1: Unlearning in Federated Online Learning to Rank. While each client performs OLTR training locally, $\{\theta_t^1, \theta_t^2, ..., \theta_t^n\}$ represent the local ranker parameters for client $\{1,2,...,N\}$ separately. $C_R$ denotes the set of remaining clients that have not request unlearning. At timestamp $t$ of unlearning, local parameters from $C_R$ will be sent to the server and contributes to the updated global ranker $\theta_{t+1}$, which will be sent back to each client of $C_R$. For clients that issue an unlearning request, their parameters will no longer be uploaded to the server, and their contributions to the server will be removed.
• Figure 2: Offline nDCG@10 obtained under three scenarios (data poisoning, model poisoning, clean) on Yahoo dataset, under three click models (Perfect, Navigational, Informational).
• Figure 3: Offline nDCG@10 obtained under clean scenario on Istella-S dataset, under three click models (Perfect, Navigational, Informational).