You Only Debias Once: Towards Flexible Accuracy-Fairness Trade-offs at Inference Time

TL;DR

This work tackles the problem of fixed accuracy-fairness trade-offs in fairness methods for high-stakes decisions. It introduces You Only Debias Once (YODO), a method that learns an objective-diverse subspace forming a line in weight space between an accuracy-optimum endpoint $\omega_1$ and a fairness-optimum endpoint $\omega_2$, parameterized by $\theta = (1-\alpha)\omega_1 + \alpha\omega_2$ for inference. By training with $\alpha \sim \mathrm{U}[0,1]$ and incorporating a cosine diversity regularizer, YODO enables arbitrary accuracy-fairness trade-offs from a single model without retraining, while supporting multiple fairness notions such as DP, EO, and Eodd. Empirical results on tabular and image data show competitive or superior Pareto fronts and smoother trade-off curves, with interpretable instance-level adjustments, demonstrating practical applicability and low overhead for real-world deployments.

Abstract

Deep neural networks are prone to various bias issues, jeopardizing their applications for high-stake decision-making. Existing fairness methods typically offer a fixed accuracy-fairness trade-off, since the weight of the well-trained model is a fixed point (fairness-optimum) in the weight space. Nevertheless, more flexible accuracy-fairness trade-offs at inference time are practically desired since: 1) stakes of the same downstream task can vary for different individuals, and 2) different regions have diverse laws or regularization for fairness. If using the previous fairness methods, we have to train multiple models, each offering a specific level of accuracy-fairness trade-off. This is often computationally expensive, time-consuming, and difficult to deploy, making it less practical for real-world applications. To address this problem, we propose You Only Debias Once (YODO) to achieve in-situ flexible accuracy-fairness trade-offs at inference time, using a single model that trained only once. Instead of pursuing one individual fixed point (fairness-optimum) in the weight space, we aim to find a "line" in the weight space that connects the accuracy-optimum and fairness-optimum points using a single model. Points (models) on this line implement varying levels of accuracy-fairness trade-offs. At inference time, by manually selecting the specific position of the learned "line", our proposed method can achieve arbitrary accuracy-fairness trade-offs for different end-users and scenarios. Experimental results on tabular and image datasets show that YODO achieves flexible trade-offs between model accuracy and fairness, at ultra-low overheads. For example, if we need $100$ levels of trade-off on the \acse dataset, YODO takes $3.53$ seconds while training $100$ fixed models consumes $425$ seconds. The code is available at https://github.com/ahxt/yodo.

Figures (26)

• Figure 1: (a):The overview of our proposed method. 2D Weight Space indicated the landscape of model accuracy and fairness. $\vcenter{}$ indicates the accuracy-optimum weight with high accuracy but low fairness, and $\vcenter{}$ indicates the fairness-optimum weight with low accuracy but high fairness. Network Subspaces shows the different subspaces correspond with different objectives (i.e., accuracy $\mathcal{L}_{ce}$ and fairness $\mathcal{L}_{f}$). (b): The loss landscape of the model accuracy (error rate) and fairness ($\Delta \text{DP}$) in the same weight space of our proposed method. The weight space is reduced to two dimensions garipov2018loss. The different points indicate different objectives, $\vcenter{}$ indicates the accuracy-optimum endpoint in the weight space, while $\vcenter{}$ indicates the fairness-optimum endpoint in the weight space. The dataset is ACS-I with gender as the sensitive attribute. $\Delta \text{DP}$ is the demographic parity difference, which asserts that the probability of a positive outcome should be the same across all demographic groups.
• Figure 2: Prediction procedure of YODO
• Figure 3: The Pareto frontier of accuracy and fairness. The first row is the fairness performance with respect to gender sensitive attribute, while the second row is race sensitive attribute. The model performance metric is Error Rate (lower is better), and the fairness metric is $\Delta \text{DP}$ (lower is better).
• Figure 4: The Pareto frontier of the model performance and fairness on the CelebA dataset. The sensitive attribute we considered is gender and age. The x-axis represents the $\Delta \text{DP}$, while the y-axis represents the error rate of the downstream task. Our proposed one-time training model achieves a comparable accuracy-fairness trade-off with that of fixed-trained models.
• Figure 5: The accuracy-fairness trade-offs at inference time with respect to $\alpha$ for three different datasets: ACS-I dataset with gender as the sensitive attribute (Left), ACS-I dataset with race as the sensitive attribute (Middle), CelebA dataset with gender as the sensitive attribute (Right). We observed that the fine-grained accuracy-fairness trade-offs could be achieved by selecting different values of $\alpha$, providing more nuanced accuracy-fairness trade-offs. Note that the results are obtained at inference time with a single trained model.