CUBIC: Concept Embeddings for Unsupervised Bias Identification using VLMs

TL;DR

CUBIC introduces an unsupervised bias-identification framework that uses Vision-Language Models to detect bias-inducing concepts without relying on labeled bias candidates or misclassified samples. It trains a linear probe on frozen image features, constructs a large concept dataset from captions, and computes a bias score $\cos{\alpha_C}$ that measures how concept presence shifts the superclass embedding toward a prediction. The method can automatically identify bias-related concepts by filtering with a zero-shot text classifier and demonstrates strong correlations with ground-truth bias across datasets, including fine-grained concepts, while outperforming or matching existing bias-discovery approaches even when no failure samples are available. This approach enables more actionable bias mitigation by revealing latent, concept-based influences on predictions and suggests avenues for broader applicability across backbones and tasks, though it remains tied to the quality of CLIP-style representations and the concept dataset coverage.

Abstract

Deep vision models often rely on biases learned from spurious correlations in datasets. To identify these biases, methods that interpret high-level, human-understandable concepts are more effective than those relying primarily on low-level features like heatmaps. A major challenge for these concept-based methods is the lack of image annotations indicating potentially bias-inducing concepts, since creating such annotations requires detailed labeling for each dataset and concept, which is highly labor-intensive. We present CUBIC (Concept embeddings for Unsupervised Bias IdentifiCation), a novel method that automatically discovers interpretable concepts that may bias classifier behavior. Unlike existing approaches, CUBIC does not rely on predefined bias candidates or examples of model failures tied to specific biases, as such information is not always available. Instead, it leverages image-text latent space and linear classifier probes to examine how the latent representation of a superclass label$\unicode{x2014}$shared by all instances in the dataset$\unicode{x2014}$is influenced by the presence of a given concept. By measuring these shifts against the normal vector to the classifier's decision boundary, CUBIC identifies concepts that significantly influence model predictions. Our experiments demonstrate that CUBIC effectively uncovers previously unknown biases using Vision-Language Models (VLMs) without requiring the samples in the dataset where the classifier underperforms or prior knowledge of potential biases.

Figures (3)

• Figure 1: CUBIC methodology illustrated. In (\ref{['fig:linear_probe']}), a linear probe classifier is constructed by training a linear SVM on the features provided by a frozen image encoder. In (\ref{['fig:angle_between_concept_and_hyperplane']}), the cosine of the angle $\alpha_{C}$ between the vector normal to the SVM hyperplane, $\vec{n}$, and the concept-driven shift in superclass embedding $f_{T}(\mathcal{L} \land C) - f_{T}(\mathcal{L})$ is calculated. Here, $f_{T}(\mathcal{L})$ represents the embedding of a superclass label common to all images (e.g., Person in the CelebA dataset liu2015faceattributes, Bird in the Waterbirds dataset sagawadistributionally). On the other hand, $\mathcal{L} \land C$ represents a prompt combining concept $C$ and its superclass label $\mathcal{L}$ (e.g., Person, Blond), and $f_{T}(\mathcal{L} \land C)$ its embedding in the latent space. In (\ref{['fig:selection_of_concepts']}), $\cos\alpha_{C}$ indicates the magnitude and the class to which concept $C$ biases the model. If $\cos\alpha_{C} > 0$, the concept-driven shift of the superclass embedding $f_{T}(\mathcal{L} \land C) - f_{T}(\mathcal{L})$ points toward the class $\bf{1}$ side of the hyperplane. This means concept $C$pushes the superclass embedding in the direction of class $\bf{1}$ (no-glasses). The opposite occurs when $\cos\alpha_{C} < 0$. Had we taken $\vec{n}$ towards class $\bf{0}$, $\cos\alpha_{C} > 0$ would indicate the concept is pushing toward class $\bf{0}$.
• Figure 2: Distribution of the Spearman correlation between ground truth concept bias $\Delta_{C}$ and CUBIC scores ($\cos{\alpha_{C}}$) across multiple CelebA-derived datasets with $\theta$-controlled undersampling. The Spearman correlation distributions demonstrate the strong predictive power of $\cos{\alpha_{C}}$ in capturing concept-induced bias variations. Our results show that CUBIC scores achieve $>0.95$ spearman correlation with $\Delta_{C}$ almost 80% of cases, validating its effectiveness as bias indicators.
• Figure 3: Images retrieved from the Waterbirds dataset sagawadistributionally evidencing the most influential biasing concepts discovered by CUBIC. CUBIC identifies finer-grained concepts beyond just forest background and water background.