Discovering Concept Directions from Diffusion-based Counterfactuals via Latent Clustering

TL;DR

CDLC addresses the need for scalable, global concept discovery in diffusion-based explanations by clustering unit latent-difference vectors between factual and counterfactual images. It reduces storage and computational overhead compared with CDCT by avoiding trajectory storage and exhaustive per-dimension searches, using spherical K-Means on a VAE-encoded latent space and CPU clustering. In ISIC dermoscopy experiments with a ResNet-50 classifier, the learned concept directions align with clinically recognized dermoscopic features and reveal potential dataset biases, demonstrating interpretability and generality across latent dimensions. The approach offers a practical, high-signal explanation tool for high-stakes domains, while acknowledging hue biases and the need for clinical validation.

Abstract

Concept-based explanations have emerged as an effective approach within Explainable Artificial Intelligence, enabling interpretable insights by aligning model decisions with human-understandable concepts. However, existing methods rely on computationally intensive procedures and struggle to efficiently capture complex, semantic concepts. This work introduces the Concept Directions via Latent Clustering (CDLC), which extracts global, class-specific concept directions by clustering latent difference vectors derived from factual and diffusion-generated counterfactual image pairs. CDLC reduces storage requirements by ~4.6% and accelerates concept discovery by ~5.3% compared to the baseline method, while requiring no GPU for clustering, thereby enabling efficient extraction of multidimensional semantic concepts across latent dimensions. This approach is validated on a real-world skin lesion dataset, demonstrating that the extracted concept directions align with clinically recognized dermoscopic features and, in some cases, reveal dataset-specific biases or unknown biomarkers. These results highlight that CDLC is interpretable, scalable, and applicable across high-stakes domains and diverse data modalities.

Figures (3)

• Figure 1: Overview of the CDLC framework. Counterfactuals are generated using a Latent Diffusion Model (LDM) with classifier guidance, following the procedure used in CDCT varshney2025generating. Factual–counterfactual image pairs are encoded using a pretrained Variational Autoencoder (VAE), and the difference between their latent representations is normalized to form unit vectors. These vectors are clustered to identify class-specific concept directions $C_j$. During inference, each direction, scaled by a factor $\alpha$, is applied to the test sample's latent representation to observe its effect on classifier output.
• Figure 2: Discovered concepts by CDLC on the ISIC dataset using the LDM encoder. Each row shows two examples: original, reconstructed, and manipulated reconstruction (left to right). The predicted probability for the target class associated with each concept direction is shown above each image.
• Figure 3: Discovered concepts by CDLC on the ISIC dataset using the CDCT encoder. Each row shows two examples: original, reconstructed, and manipulated images (left to right). The predicted probability for the target class associated with each concept direction is shown above each image.