Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Fairness Vision-Language Learning for Medical Image Analysis

Sparsh Bansal, Mingyang Wu, Xin Wang, Shu Hu

TL;DR

This work tackles fairness and robustness in vision-language models for medical image analysis by introducing RobustFairVLM, a training framework that combines Dynamic Bad Pair Mining (DBPM) to downweight faulty image–text pairs and a Sinkhorn-distance-based fairness term to align distributions across protected attributes. Built on CLIP-style CLIP ViT-B/16 (and evaluated with BLIP-2 to demonstrate generalizability), the method optimizes a sequence of losses ($L_1$, $L_2$, $L_3$) that progressively downweight noisy data and enforce demographic balance. Empirical results on the Harvard-FairVLMed glaucoma dataset show significant gains: RobustFairCLIP improves total AUC by up to 8.6% over CLIP and 3% over FairCLIP, with comparable gains observed for RobustFairBLIP-2 on BLIP-2 architectures. The framework enables scalable, multi-modal medical diagnosis with improved fairness and resilience to data imperfections, though future work should extend fairness regularization to multiple protected attributes and additional architectures.

Abstract

The advent of Vision-Language Models (VLMs) in medical image analysis has the potential to help process multimodal inputs and increase performance over traditional inference methods. However, when considering the domain in which these models will be implemented, fairness and robustness are important to ensure the model stays true for any patient. In this paper, we introduce a framework for ensuring robustness and fairness of VLM models. This framework modifies the loss function at training by identifying and adjusting faulty image-text pairs through a Dynamic Bad Pair Mining algorithm and also utilizing Sinkhorn distance to ensure the loss distributions of protected groups do not deviate from the total loss. Experimental testing of our framework shows up to a 8.6\% improvement when looking at equity-scaled AUC.

Robust Fairness Vision-Language Learning for Medical Image Analysis

TL;DR

This work tackles fairness and robustness in vision-language models for medical image analysis by introducing RobustFairVLM, a training framework that combines Dynamic Bad Pair Mining (DBPM) to downweight faulty image–text pairs and a Sinkhorn-distance-based fairness term to align distributions across protected attributes. Built on CLIP-style CLIP ViT-B/16 (and evaluated with BLIP-2 to demonstrate generalizability), the method optimizes a sequence of losses (, , ) that progressively downweight noisy data and enforce demographic balance. Empirical results on the Harvard-FairVLMed glaucoma dataset show significant gains: RobustFairCLIP improves total AUC by up to 8.6% over CLIP and 3% over FairCLIP, with comparable gains observed for RobustFairBLIP-2 on BLIP-2 architectures. The framework enables scalable, multi-modal medical diagnosis with improved fairness and resilience to data imperfections, though future work should extend fairness regularization to multiple protected attributes and additional architectures.

Abstract

The advent of Vision-Language Models (VLMs) in medical image analysis has the potential to help process multimodal inputs and increase performance over traditional inference methods. However, when considering the domain in which these models will be implemented, fairness and robustness are important to ensure the model stays true for any patient. In this paper, we introduce a framework for ensuring robustness and fairness of VLM models. This framework modifies the loss function at training by identifying and adjusting faulty image-text pairs through a Dynamic Bad Pair Mining algorithm and also utilizing Sinkhorn distance to ensure the loss distributions of protected groups do not deviate from the total loss. Experimental testing of our framework shows up to a 8.6\% improvement when looking at equity-scaled AUC.
Paper Structure (19 sections, 11 equations, 2 figures, 2 tables)

This paper contains 19 sections, 11 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Models in Medical Image Analysis
  4. Robust VLMs in Medical Image Analysis
  5. Fair VLMs in Medical Image Analysis
  6. Method
  7. Overview
  8. One-shot Contrastive Learning with CLIP
  9. Dynamic Bad Pair Mining
  10. Fair Learning
  11. Pre-trained Models
  12. Experiments
  13. Experimental Settings
  14. Dataset
  15. Evaluation Metrics
  16. ...and 4 more sections

Figures (2)

  • Figure 1: A few examples from the Harvard-FairVLMed database luo2024fairclip that is used for training and fairness optimization, showing SLO Fundus photography images for glaucoma diagnosis.
  • Figure 2: Overview of our proposed model using CLIP ViT-B/16 dosovitskiy2020image as the base model with the loss function algorithms to account for erroneous pairs through Bad Pair Mining and equity through the Sinkhorn distance