Threshold-Independent Fair Matching through Score Calibration
Mohammad Hossein Moslemi, Mostafa Milani
TL;DR
This work tackles threshold-dependent fairness in entity matching by introducing Distributional Parity (DSP), a threshold-agnostic fairness criterion that extends DP, EO, and EOD across score distributions. It proposes a post-hoc score calibration using Wasserstein barycenters to align minority and majority score distributions toward a central barycenter, with a geometric repair $s_\lambda=(1-\lambda)s+\lambda\hat{s}$ that minimizes DSP for $\text{TPR}$ and $\text{FPR}$ while preserving $\text{AUC}$. Experiments across four EM methods and four benchmarks show DSP reveals biases not detected by $\Delta$xAUC, and calibration reduces DSP disparities with only modest loss in accuracy. The approach offers a practical, retraining-free pathway to fair EM and highlights directions for stronger guarantees and alternative calibration techniques.
Abstract
Entity Matching (EM) is a critical task in numerous fields, such as healthcare, finance, and public administration, as it identifies records that refer to the same entity within or across different databases. EM faces considerable challenges, particularly with false positives and negatives. These are typically addressed by generating matching scores and apply thresholds to balance false positives and negatives in various contexts. However, adjusting these thresholds can affect the fairness of the outcomes, a critical factor that remains largely overlooked in current fair EM research. The existing body of research on fair EM tends to concentrate on static thresholds, neglecting their critical impact on fairness. To address this, we introduce a new approach in EM using recent metrics for evaluating biases in score based binary classification, particularly through the lens of distributional parity. This approach enables the application of various bias metrics like equalized odds, equal opportunity, and demographic parity without depending on threshold settings. Our experiments with leading matching methods reveal potential biases, and by applying a calibration technique for EM scores using Wasserstein barycenters, we not only mitigate these biases but also preserve accuracy across real world datasets. This paper contributes to the field of fairness in data cleaning, especially within EM, which is a central task in data cleaning, by promoting a method for generating matching scores that reduce biases across different thresholds.