Mitigating Group-Level Fairness Disparities in Federated Visual Language Models

TL;DR

The paper tackles group fairness in federated visual-language models (VLMs) by introducing FVL-FP, a parameter-efficient framework that combines fair prompt tuning with three mechanisms: Cross-Layer Demographic Fair Prompting (CDFP) to neutralize bias in embeddings, Demographic Subspace Orthogonal Projection (DSOP) to remove demographic information from visual representations, and Fair-aware Prompt Fusion (FPF) to balance contributions from diverse clients during aggregation. The approach formalizes fairness in FL-VLMs via an equal-opportunity metric $F_{global}$ and a joint objective that couples the task loss with a fairness term, enabling effective debiasing under non-IID data. Empirical results on CelebA and FairFace across smiling and age-detection tasks show FVL-FP reduces demographic disparity by about 45% on average while maintaining accuracy within approximately 6% of state-of-the-art, with minimal computational overhead and strong robustness to varying federation scales. The work offers a practical, privacy-preserving solution for equitable multimodal systems in federated settings, advancing fair deployment of VLMs in real-world, distributed environments.

Abstract

Visual language models (VLMs) have shown remarkable capabilities in multimodal tasks but face challenges in maintaining fairness across demographic groups, particularly when deployed in federated learning (FL) environments. This paper addresses the critical issue of group fairness in federated VLMs by introducing FVL-FP, a novel framework that combines FL with fair prompt tuning techniques. We focus on mitigating demographic biases while preserving model performance through three innovative components: (1) Cross-Layer Demographic Fair Prompting (CDFP), which adjusts potentially biased embeddings through counterfactual regularization; (2) Demographic Subspace Orthogonal Projection (DSOP), which removes demographic bias in image representations by mapping fair prompt text to group subspaces; and (3) Fair-aware Prompt Fusion (FPF), which dynamically balances client contributions based on both performance and fairness metrics. Extensive evaluations across four benchmark datasets demonstrate that our approach reduces demographic disparity by an average of 45\% compared to standard FL approaches, while maintaining task performance within 6\% of state-of-the-art results. FVL-FP effectively addresses the challenges of non-IID data distributions in federated settings and introduces minimal computational overhead while providing significant fairness benefits. Our work presents a parameter-efficient solution to the critical challenge of ensuring equitable performance across demographic groups in privacy-preserving multimodal systems.

Figures (4)

• Figure 1: Federated Vision-Language Models. For each node, clients fine-tune local vision-language models based on local datasets. However, local datasets contain underlying biases in group fairness, which affect global group fairness. In this work, we propose a federated vision-language model framework that eliminates bias using fairness prompt tuning.
• Figure 2: The framework of FVL-FP. For each node, clients optimize for fairness through CDFP and DSOP algorithms. Following the local training phase, clients transmit local fairness prompts to the central server, which then performs fairness prompt aggregation using the FPF algorithm to construct a global fair prompt.
• Figure 3: Cross-Layer Demographic Fair Prompting Algorithm. Our method inserts demographic fair prompts at the embedding layer and propagates them through transformer layers with adaptive dynamic residual connections. The GAP mechanism enables learnable cross-layer connections for effective bias mitigation while preserving model performance.
• Figure 4: Demographic Subspace Orthogonal Projection. We orthogonally project visual representations away from demographic subspaces to reduce bias while preserving task-relevant information.