humancompatible.detect: a Python Toolkit for Detecting Bias in AI Models

TL;DR

This work tackles the challenge of detecting and evaluating bias in AI, with a focus on intersectional bias in tabular data. It introduces two complementary methods—Maximum Subgroup Discrepancy (MSD) and a subsampled distance test—implemented in the Python toolkit humancompatible.detect, aiming for scalable and interpretable bias analysis. MSD identifies the most discrepant subgroup across all protected attribute intersections, leveraging optimization and offering linear sample complexity in the number of protected attributes, while the subsampled test provides threshold-based bias checks with PAC-style guarantees for individual subgroups. The open-source toolkit, licensed under Apache-2.0, supports multiple input modalities and a user-friendly API to aid regulatory compliance and practical fairness auditing in high-risk AI deployments.

Abstract

There is a strong recent emphasis on trustworthy AI. In particular, international regulations, such as the AI Act, demand that AI practitioners measure data quality on the input and estimate bias on the output of high-risk AI systems. However, there are many challenges involved, including scalability (MMD) and computability (Wasserstein-1) issues of traditional methods for estimating distances on measure spaces. Here, we present humancompatible.detect, a toolkit for bias detection that addresses these challenges. It incorporates two newly developed methods to detect and evaluate bias: maximum subgroup discrepancy (MSD) and subsampled $\ell_\infty$ distances. It has an easy-to-use API documented with multiple examples. humancompatible.detect is licensed under the Apache License, Version 2.0.

Figures (5)

• Figure 1: A diagram of possible uses of the humancompatible.detect toolkit. In addition to evaluating bias of a protected group, it allows for detection of the most biased subgroup given the dataset.
• Figure 2: detect_bias activity diagram. Depending on the input mode used, different functions can be run, always calling most_biased_subgroup. After that, the data preprocessing takes places with prepare_dataset. Subsequently, the program stops if a method different from MSD has been selected, otherwise outputs the most affected subgroup as a result. This allows for easy extensibility. More details on the prepare_dataset are in Figure \ref{['fig:prepare_diag']}.
• Figure 3: evaluate_bias activity diagram. Both input and preprocessing are identical to detect_bias, with the addition of keyword arguments specifying the (sub)group (Group definition) and potentially other arguments (e.g., the $\Delta$ threshold). The output can either be a value (e.g., subgroup discrepancy) or a boolean value whether bias is above a given threshold (e.g., in subsampled $\ell_\infty$). More details on the prepare_dataset are in Figure \ref{['fig:prepare_diag']}.
• Figure 4: prepare_dataset activity diagram. This function preprocesses the input data in the form of a pandas DataFrame, y, being the target column and X, the rest of the data. It prunes incomplete rows, allows for binarization, and also for specialized feature processing, such as mapping categorical variables into variables with fewer distinct values or custom binning of continuous variables. It also allows for subsampling the data, if passed data is too large. This can speed up the processing without reducing accuracy too much, since implemented methods have low sample complexity.
• Figure 5: An illustration of why evaluating marginal bias does not suffice. A classifier accepts the same proportion of each age and race group, but rejects the entire subgroup (group intersection) of young blue people. This bias would go undetected unless group intersections are considered. The graphic is a modification of Figure 1 in nemecek_bias_2025.