Using Self-supervised Learning Can Improve Model Fairness

TL;DR

This work investigates whether self-supervised learning (SSL) can enhance model fairness in human-centric, multimodal data. By proposing a five-stage fairness assessment framework and evaluating a SimCLR-style SSL setup with gradual unfreezing across three real-world datasets (MIMIC, MESA, GLOBEM), the study demonstrates that SSL can achieve competitive accuracy while delivering meaningful improvements in fairness metrics compared to fully supervised baselines. The analysis links fairness gains to representation differences across protected attributes via conditioned $CKA$, showing that larger performance gaps between groups align with larger latent disparities. The findings suggest SSL can serve as an implicit bias-mitigation step in healthcare-like tasks, especially when data or labeling resources are limited, though not a panacea for all fairness challenges. Overall, SSL offers a practical path to more equitable predictions when carefully combined with domain-specific evaluation and bias-awareness during fine-tuning.

Abstract

Self-supervised learning (SSL) has become the de facto training paradigm of large models, where pre-training is followed by supervised fine-tuning using domain-specific data and labels. Despite demonstrating comparable performance with supervised methods, comprehensive efforts to assess SSL's impact on machine learning fairness (i.e., performing equally on different demographic breakdowns) are lacking. Hypothesizing that SSL models would learn more generic, hence less biased representations, this study explores the impact of pre-training and fine-tuning strategies on fairness. We introduce a fairness assessment framework for SSL, comprising five stages: defining dataset requirements, pre-training, fine-tuning with gradual unfreezing, assessing representation similarity conditioned on demographics, and establishing domain-specific evaluation processes. We evaluate our method's generalizability on three real-world human-centric datasets (i.e., MIMIC, MESA, and GLOBEM) by systematically comparing hundreds of SSL and fine-tuned models on various dimensions spanning from the intermediate representations to appropriate evaluation metrics. Our findings demonstrate that SSL can significantly improve model fairness, while maintaining performance on par with supervised methods-exhibiting up to a 30% increase in fairness with minimal loss in performance through self-supervision. We posit that such differences can be attributed to representation dissimilarities found between the best- and the worst-performing demographics across models-up to x13 greater for protected attributes with larger performance discrepancies between segments.

Figures (12)

• Figure 1: Overview of the proposed fairness assessment framework for SSL. Starting with benchmark selection, we systematically study the impact of fine-tuning on fairness through a novel combination of evaluation and representation learning metrics.
• Figure 2: MIMIC
• Figure 3: MESA
• Figure 4: GLOBEM
• Figure 6: The relationship between segment size and performance (AUC-ROC) across datasets. The smaller the segment the larger the performance discrepancies. Fitted lowess curves show that SSL lies closer to the "fair" (dashed) line.