A Closed-Form Solution for Debiasing Vision-Language Models with Utility Guarantees Across Modalities and Tasks

Abstract

While Vision-Language Models (VLMs) have achieved remarkable performance across diverse downstream tasks, recent studies have shown that they can inherit social biases from the training data and further propagate them into downstream applications. To address this issue, various debiasing approaches have been proposed, yet most of them aim to improve fairness without having a theoretical guarantee that the utility of the model is preserved. In this paper, we introduce a debiasing method that yields a \textbf{closed-form} solution in the cross-modal space, achieving Pareto-optimal fairness with \textbf{bounded utility losses}. Our method is \textbf{training-free}, requires \textbf{no annotated data}, and can jointly debias both visual and textual modalities across downstream tasks. Extensive experiments show that our method outperforms existing methods in debiasing VLMs across diverse fairness metrics and datasets for both group and \textbf{intersectional} fairness in downstream tasks such as zero-shot image classification, text-to-image retrieval, and text-to-image generation while preserving task performance.

Figures (12)

• Figure 1: (a) Previous methods such as PRISM-mini molahasani2025prism and Orth-Proj chuang2023debiasing debias through an orthogonal projection to the subspace $\mathcal{S}$ spanned by the embeddings of group-explicit prompts (group prototypes) such as "a photo of a female/male doctor." However, the subspace $\mathcal{S}$ contains not only attribute-related information but also essential semantic content (e.g., "doctor") that we wish to preserve. (b) We instead debias through an orthogonal projection to the attribute subspace $\mathcal{A}$ so that $\vec{u}$ can effectively removes bias while maintaining content-related semantics.
• Figure 2: Illustration of the group prototype construction process.
• Figure 3: Illustration of the optimal solution space of $\vec{u}$.
• Figure 4: Illustrative examples for $o=\text{"doctor"}$. We randomly sample ten generated images for each method. Female-looking samples are marked in red and numbered. On the left, a more balanced ratio of female- and male-looking samples indicates lower bias, while on the right, fewer female-looking samples reflect better preservation of self-utility (see Appendix G for more examples with other occupations).
• Figure 5: An illustration of the prompts we use in an LLM to insert the group specification and generate the variants for each group $g$.