Local Concept Embeddings for Analysis of Concept Distributions in Vision DNN Feature Spaces
Georgii Mikriukov, Gesina Schwalbe, Korinna Bade
TL;DR
This work introduces Local Concept Embeddings (LoCEs) to capture the distribution of concept representations in vision DNN latent spaces, moving beyond traditional single-vector concept embeddings. By computing per-image LoCEs and analyzing their multi-modal distributions with hierarchical clustering and UMAP-GMM, the authors unveil sub-concepts, concept confusion, and concept-level outliers. LoCEs enable new debugging and retrieval applications, including sub-concept discovery, quantifying concept separation, and context-aware image retrieval, while maintaining competitive concept segmentation performance. Across MS COCO, PASCAL VOC, and Capybara, LoCEs reveal that concept representations are complex and context-dependent, highlighting limitations of global C-XAI baselines and offering a principled framework for distribution-focused explainability. The work lays a foundation for using LoCEs to diagnose model behavior, guide data improvements, and inform architectural design, with future directions including multi-layer analyses and backbone diversification.
Abstract
Insights into the learned latent representations are imperative for verifying deep neural networks (DNNs) in critical computer vision (CV) tasks. Therefore, state-of-the-art supervised Concept-based eXplainable Artificial Intelligence (C-XAI) methods associate user-defined concepts like ``car'' each with a single vector in the DNN latent space (concept embedding vector). In the case of concept segmentation, these linearly separate between activation map pixels belonging to a concept and those belonging to background. Existing methods for concept segmentation, however, fall short of capturing implicitly learned sub-concepts (e.g., the DNN might split car into ``proximate car'' and ``distant car''), and overlap of user-defined concepts (e.g., between ``bus'' and ``truck''). In other words, they do not capture the full distribution of concept representatives in latent space. For the first time, this work shows that these simplifications are frequently broken and that distribution information can be particularly useful for understanding DNN-learned notions of sub-concepts, concept confusion, and concept outliers. To allow exploration of learned concept distributions, we propose a novel local concept analysis framework. Instead of optimizing a single global concept vector on the complete dataset, it generates a local concept embedding (LoCE) vector for each individual sample. We use the distribution formed by LoCEs to explore the latent concept distribution by fitting Gaussian mixture models (GMMs), hierarchical clustering, and concept-level information retrieval and outlier detection. Despite its context sensitivity, our method's concept segmentation performance is competitive to global baselines. Analysis results are obtained on three datasets and six diverse vision DNN architectures, including vision transformers (ViTs).