Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Configurable Fairness: Direct Optimization of Parity Metrics via Vision-Language Models

Miao Zhang, Rumi Chunara

TL;DR

The paper tackles fairness gaps in image recognition arising from imbalanced groups and unknown sensitive attributes. It proposes a configurable fairness framework that directly optimizes parity-based metrics using vision-language model derived attribute relevancy as soft regularizers, enabling equality-of-opportunity, equalized odds, or accuracy parity without group labels. The approach yields differentiable loss terms that connect to each metric via PCC-based penalties, and experiments across CelebA, UTKFace, and Dogs & Cats show improved parity with robust performance under noisy attribute signals. This method offers a flexible, privacy-preserving path to fairness in vision tasks and highlights the ongoing trade-offs between different parity notions in real-world deployments.

Abstract

Performance disparities of image recognition across demographic groups are known to exist in deep learning-based models, due to imbalanced group representations or spurious correlation between group and target labels. Previous work has addressed such challenges without relying on expensive group labels, typically by upweighting high-loss samples or balancing discovered clusters. However, these heuristic strategies lack direct connection to specific fairness metrics and cannot guarantee optimization of parity-based criteria like equal opportunity, which ensures equal chance to receive positive outcomes across groups. In this work, we propose a novel paradigm that directly optimizes parity-based fairness metrics through specifically designed training objectives, without requiring group labels. We leverage vision-language models to analyze sensitive attribute relevancy for individual samples, then formulate loss functions that mathematically connect to each target fairness metric. This enables flexible optimization of different fairness criteria based on application needs. Experiments on multiple image classification datasets show that our metric-specific approach significantly improves parity-based fairness criteria and outperforms existing methods.

Configurable Fairness: Direct Optimization of Parity Metrics via Vision-Language Models

TL;DR

The paper tackles fairness gaps in image recognition arising from imbalanced groups and unknown sensitive attributes. It proposes a configurable fairness framework that directly optimizes parity-based metrics using vision-language model derived attribute relevancy as soft regularizers, enabling equality-of-opportunity, equalized odds, or accuracy parity without group labels. The approach yields differentiable loss terms that connect to each metric via PCC-based penalties, and experiments across CelebA, UTKFace, and Dogs & Cats show improved parity with robust performance under noisy attribute signals. This method offers a flexible, privacy-preserving path to fairness in vision tasks and highlights the ongoing trade-offs between different parity notions in real-world deployments.

Abstract

Performance disparities of image recognition across demographic groups are known to exist in deep learning-based models, due to imbalanced group representations or spurious correlation between group and target labels. Previous work has addressed such challenges without relying on expensive group labels, typically by upweighting high-loss samples or balancing discovered clusters. However, these heuristic strategies lack direct connection to specific fairness metrics and cannot guarantee optimization of parity-based criteria like equal opportunity, which ensures equal chance to receive positive outcomes across groups. In this work, we propose a novel paradigm that directly optimizes parity-based fairness metrics through specifically designed training objectives, without requiring group labels. We leverage vision-language models to analyze sensitive attribute relevancy for individual samples, then formulate loss functions that mathematically connect to each target fairness metric. This enables flexible optimization of different fairness criteria based on application needs. Experiments on multiple image classification datasets show that our metric-specific approach significantly improves parity-based fairness criteria and outperforms existing methods.
Paper Structure (21 sections, 10 equations, 1 figure, 7 tables)

This paper contains 21 sections, 10 equations, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Fairness in Image Recognition
  4. Leveraging Vision-Language Models
  5. Fairness Problem Formulation
  6. Why Configurable Fairness Matters
  7. Method
  8. Preliminaries
  9. Optimizing Equal Opportunity
  10. Optimizing Equalized Odds
  11. Optimizing Accuracy Parity
  12. Experiment
  13. Datasets
  14. Baselines and Implementation
  15. Additional Analysis
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Comparing the average group level test accuracy of ERM method, and our proposed methods optimizing for equal opportunity $\mathcal{L}_{EOp}$, equalized odds $\mathcal{L}_{EOd}$, and accuracy parity $\mathcal{L}_{AP}$, across six attribute recognition tasks on the CelebA dataset.