BendVLM: Test-Time Debiasing of Vision-Language Embeddings

TL;DR

Bend-VLM is proposed, a nonlinear, fine-tuning-free approach for VLM embedding debiasing that tailors the debiasing operation to each unique input, making this method more appropriate for online, open-set tasks such as retrieval and text guided image generation.

Abstract

Vision-language model (VLM) embeddings have been shown to encode biases present in their training data, such as societal biases that prescribe negative characteristics to members of various racial and gender identities. VLMs are being quickly adopted for a variety of tasks ranging from few-shot classification to text-guided image generation, making debiasing VLM embeddings crucial. Debiasing approaches that fine-tune the VLM often suffer from catastrophic forgetting. On the other hand, fine-tuning-free methods typically utilize a "one-size-fits-all" approach that assumes that correlation with the spurious attribute can be explained using a single linear direction across all possible inputs. In this work, we propose Bend-VLM, a nonlinear, fine-tuning-free approach for VLM embedding debiasing that tailors the debiasing operation to each unique input. This allows for a more flexible debiasing approach. Additionally, we do not require knowledge of the set of inputs a priori to inference time, making our method more appropriate for online, open-set tasks such as retrieval and text guided image generation.

Figures (2)

• Figure 1: Overview of our two-step Bend-VLM method. In this example, the initial query embedding of doctor is more strongly associated with males, and the CCF distance is $0.10$. After performing debiasing step 1, Orthogonalizing the Embedding, the embedding is modified to remove bias along the gender direction defined by "male doctor" and "female doctor". This still results in a CCF distance of $0.05$. We then perform the second debiasing step, where the query embedding is again modified to be equidistant to the relevant male and female images. The final representation achieves the optimal distance of $\boldsymbol{0.00}$.
• Figure 2: Our approach increases accuracy while decreasing bias.

Theorems & Definitions (4)

• lemma 1: Orthogonalization does not yield Class Conditional Fairness.
• lemma 2
• proof : Proof of Lemma \ref{['lm:ortho']}.
• proof : Proof of Lemma \ref{['lm:const_solution']}