FairDiverse: A Comprehensive Toolkit for Fair and Diverse Information Retrieval Algorithms

TL;DR

FairDiverse addresses the need for standardized, transparent evaluation of fairness and diversity in information retrieval by delivering an open-source toolkit that integrates pre-, in-, and post-processing methods across both search and recommendation. It encompasses 28 fairness- and diversity-aware models applied to 16 base IR models, with datasets and metrics that enable comprehensive benchmarking. The framework emphasizes extensibility and reproducibility through flexible APIs and concrete usage workflows, while providing detailed analyses of benchmark results to illuminate trade-offs between accuracy and fairness/diversity. The work advances the IR community by offering a unified platform for fair comparisons, benchmarks, and customization, with future directions including dynamic fairness settings and broader beyond-accuracy objectives.

Abstract

In modern information retrieval (IR). achieving more than just accuracy is essential to sustaining a healthy ecosystem, especially when addressing fairness and diversity considerations. To meet these needs, various datasets, algorithms, and evaluation frameworks have been introduced. However, these algorithms are often tested across diverse metrics, datasets, and experimental setups, leading to inconsistencies and difficulties in direct comparisons. This highlights the need for a comprehensive IR toolkit that enables standardized evaluation of fairness- and diversity-aware algorithms across different IR tasks. To address this challenge, we present FairDiverse, an open-source and standardized toolkit. FairDiverse offers a framework for integrating fair and diverse methods, including pre-processing, in-processing, and post-processing techniques, at different stages of the IR pipeline. The toolkit supports the evaluation of 28 fairness and diversity algorithms across 16 base models, covering two core IR tasks (search and recommendation) thereby establishing a comprehensive benchmark. Moreover, FairDiverse is highly extensible, providing multiple APIs that empower IR researchers to swiftly develop and evaluate their own fairness and diversity aware models, while ensuring fair comparisons with existing baselines. The project is open-sourced and available on https://github.com/XuChen0427/FairDiverse.

Figures (3)

• Figure 1: Overall architecture of FairDiverse. We categorize fairness- and diversity-aware algorithms into pre-processing, in-processing, and post-processing stages, corresponding to data processing, model training, and result evaluation phases of IR.
• Figure 2: The usage of FairDiverse with three steps: (1) Download the datasets and check the default parameters of the four stages of pipelines; (2) Set custom configuration file to execute the pipeline. The $*$tuning_variables allow you to define variable values for the default settings across the four pipeline stages, with the In-processing configuration file overriding these default values when specified; (3) Run the shell command, with the task, stage, dataset, and your custom configuration file.
• Figure 3: The custom steps for fairness and diversity-aware search and recommender models named YourModel. The differently colored areas indicate the code you need to add when developing different types of model. Generally, you can follow three steps: (1) define custom model parameters, (2) develop your model based on its type, and (3) integrate it into the pipeline.