Conceptual Learning via Embedding Approximations for Reinforcing Interpretability and Transparency

TL;DR

This study proposes Conceptual Learning via Embedding Approximations for Reinforcing Interpretability and Transparency, abbreviated as CLEAR, a framework for constructing a CBM for image classification by approximate the embedding of concepts within the latent space of a vision-language model (VLM) by learning the scores associated with the joint distribution of images and concepts.

Abstract

Concept bottleneck models (CBMs) have emerged as critical tools in domains where interpretability is paramount. These models rely on predefined textual descriptions, referred to as concepts, to inform their decision-making process and offer more accurate reasoning. As a result, the selection of concepts used in the model is of utmost significance. This study proposes \underline{\textbf{C}}onceptual \underline{\textbf{L}}earning via \underline{\textbf{E}}mbedding \underline{\textbf{A}}pproximations for \underline{\textbf{R}}einforcing Interpretability and Transparency, abbreviated as CLEAR, a framework for constructing a CBM for image classification. Using score matching and Langevin sampling, we approximate the embedding of concepts within the latent space of a vision-language model (VLM) by learning the scores associated with the joint distribution of images and concepts. A concept selection process is then employed to optimize the similarity between the learned embeddings and the predefined ones. The derived bottleneck offers insights into the CBM's decision-making process, enabling more comprehensive interpretations. Our approach was evaluated through extensive experiments and achieved state-of-the-art performance on various benchmarks. The code for our experiments is available at https://github.com/clearProject/CLEAR/tree/main

Figures (9)

• Figure 1: The core components of CLEAR, our proposed paradigm for constructing a data-adaptive CBM by modeling the joint distribution of images and concepts.
• Figure 2: An overview of CLEAR. In step 1 we obtain the image and descriptor embeddings and train the score network. In step 2 we learn the concept approximations and in step 3 we obtain the approximation-description similarity matrix, select the concepts by finding the optimal allocation and integrate our bottleneck.
• Figure 3: Test accuracy comparison on different bottleneck sizes across all datasets.
• Figure 4: Interpretability analysis of our proposed framework during inference
• Figure 5: t-SNE visualization of CIFAR-10 descriptors