Toward Fairness via Maximum Mean Discrepancy Regularization on Logits Space

TL;DR

This work tackles fairness in high-risk computer-vision tasks by addressing Equalized Odds through a logits-space regularizer. The authors introduce Logits-MMD, which minimizes the Maximum Mean Discrepancy between the logit distributions of different sensitive groups for each class, integrated with standard cross-entropy loss as $\min_{\Theta} L_{CE}(\Theta) + \lambda L_{MMD}(\Theta)$. They argue that prior logits-space methods (Gaussian Assumption and Histogram Approximation) impose distributional priors that misalign with EO, and demonstrate that MMD provides a principled, threshold-free alignment via RKHS with a Gaussian kernel. Empirically, Logits-MMD achieves state-of-the-art equalized odds performance on CelebA and UTK Face, and generalizes to bias scenarios in Dogs and Cats, confirming its robustness and practical impact for fair facial attribute classification. The approach offers a principled, scalable path to fair predictions without onerous distributional assumptions, with broad applicability to multi-attribute fairness in vision systems.

Abstract

Fairness has become increasingly pivotal in machine learning for high-risk applications such as machine learning in healthcare and facial recognition. However, we see the deficiency in the previous logits space constraint methods. Therefore, we propose a novel framework, Logits-MMD, that achieves the fairness condition by imposing constraints on output logits with Maximum Mean Discrepancy. Moreover, quantitative analysis and experimental results show that our framework has a better property that outperforms previous methods and achieves state-of-the-art on two facial recognition datasets and one animal dataset. Finally, we show experimental results and demonstrate that our debias approach achieves the fairness condition effectively.

Figures (4)

• Figure 1: The Logits-MMD constraint pulls the logits distribution of different sensitive groups together i.e. A = 0 and A = 1. (a) is an example of logits distribution without Logits-MMD constraint. (b) is an example of imposing Logits-MMD constraint.
• Figure 2: The illustrative concept of Logits-MMD. $L_{MMD}$ minimizes the logits distribution discrepancy between each sensitive group for the same class and achieves fairness. $L_{CE}$ is the cross-entropy loss, which enforces the model in achieving high predictive accuracy
• Figure 3: The result of GA and MMD fitting a toy example of a multimodal distribution. MMD could fit well on the ground truth toy example, while GA cannot. The x-axis is the value of the data points from the toy example, and the y-axis is the density.
• Figure 4: The first row is the confidence score PDF with the ground truth as $Big \ Nose$ (B), and the second is the ground truth as $Not \ Big \ Nose$ (NB). From left to right are Logits-MMD, CNN, Gaussian Assumption (GA), and Histogram Approximation (HA), respectively."T" refers to the target attribute, and "S" to the sensitive attribute. The density estimation will smooth the curve at the max and min points, so there will be values that exceed 1 and less than 0.