Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Attention-based Framework for Fair Contrastive Learning

Stefan K. Nielsen, Tan M. Nguyen

TL;DR

A new method for fair contrastive learning that employs an attention mechanism to model bias-causing interactions, enabling the learning of a fairer and semantically richer embedding space and can significantly boost bias removal from learned representations without compromising downstream accuracy.

Abstract

Contrastive learning has proven instrumental in learning unbiased representations of data, especially in complex environments characterized by high-cardinality and high-dimensional sensitive information. However, existing approaches within this setting require predefined modelling assumptions of bias-causing interactions that limit the model's ability to learn debiased representations. In this work, we propose a new method for fair contrastive learning that employs an attention mechanism to model bias-causing interactions, enabling the learning of a fairer and semantically richer embedding space. In particular, our attention mechanism avoids bias-causing samples that confound the model and focuses on bias-reducing samples that help learn semantically meaningful representations. We verify the advantages of our method against existing baselines in fair contrastive learning and show that our approach can significantly boost bias removal from learned representations without compromising downstream accuracy.

An Attention-based Framework for Fair Contrastive Learning

TL;DR

A new method for fair contrastive learning that employs an attention mechanism to model bias-causing interactions, enabling the learning of a fairer and semantically richer embedding space and can significantly boost bias removal from learned representations without compromising downstream accuracy.

Abstract

Contrastive learning has proven instrumental in learning unbiased representations of data, especially in complex environments characterized by high-cardinality and high-dimensional sensitive information. However, existing approaches within this setting require predefined modelling assumptions of bias-causing interactions that limit the model's ability to learn debiased representations. In this work, we propose a new method for fair contrastive learning that employs an attention mechanism to model bias-causing interactions, enabling the learning of a fairer and semantically richer embedding space. In particular, our attention mechanism avoids bias-causing samples that confound the model and focuses on bias-reducing samples that help learn semantically meaningful representations. We verify the advantages of our method against existing baselines in fair contrastive learning and show that our approach can significantly boost bias removal from learned representations without compromising downstream accuracy.

Paper Structure

This paper contains 22 sections, 2 theorems, 24 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Conditional Contrastive Learning
  4. Attention Mechanism
  5. Fairness Meets Attention
  6. FARE: Fairness-Aware Attention
  7. SparseFARE: Sparse Fairness-Aware Attention
  8. FAREContrast: Fair Attention-Contrastive Criterion for Contrastive Learning
  9. Experiments
  10. Related Work
  11. Concluding Remarks
  12. Experimental Details
  13. Training and Evaluation
  14. Baselines
  15. Connection between Kernel-based Scoring Function Estimation in tsai2022conditional and Attention
  16. ...and 7 more sections

Key Result

Proposition 1

Given $\{x_i, y_i, z_i\}_{i=1}^{b} \sim P^{\otimes b}_{XYZ}$, the finite-sample estimation of $e^{f(x_i, y)}$ is $\sum_{j=1}^{b}\text{softmax}\left((W_Qz_i)^{\top}W_Kz_j/\rho\right) \left[\phi (g_{\theta_{X}}(x_i))^{\top}\phi(g_{\theta_{Y}}(y_j))\right]$, which is the output of an attention mechanis

Figures (3)

  • Figure 1: Sparse Fair-Aware Attention (SparseFARE) using LSH to discard bias-causing samples. Relative to the anchor's protected attribute status (blue), the fairness-aware attention (FARE) first groups the samples according to their bias attribute and discards any samples that are likely to be highly bias-inducing (brown). Attention scores between similar and bias-reducing samples are then computed.
  • Figure 2: colorMNIST dataset tsai2022conditional
  • Figure 3: Fairness-Accuracy Tradeoff of SparseFARE and CCLK-

Theorems & Definitions (8)

  • Proposition 1: Conditional Estimation of $e^{f(x_i, y)}$ when $y \sim P_{Y|Z = z_i}$
  • Definition 1: Fairness-Aware Attention
  • Remark 1
  • Remark 2
  • Definition 2: Fairness-Aware Attention with Sparsification
  • Proposition 2: Kernel-Based Scoring Function Estimation tsai2022conditional
  • proof : Proof of kernel-based scoring function estimation
  • Definition 3