Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning

TL;DR

The paper tackles the interpretability gap in high-performance image classification by building Concept Bottleneck Models (CBMs) on top of pre-trained multi-modal encoders like CLIP. It introduces three training variants—Sparse-CBM (Gumbel-Softmax sparsity), Contrastive-CBM, and $\ell_1$-CBM—along with a no-training Concept Matrix Search (CMS) to enhance predictions using latent space similarities. A practical framework is provided for automatic concept generation and a two-layer bottleneck design (CBL and FC) trained with two optimizers, achieving improved accuracy across CIFAR-10/100, ImageNet, and CUB200 while preserving interpretability. The work advances scalable, explainable CBMs for CLIP-like models and offers insights into how sparse concept activations relate to robust classification across benchmarks.

Abstract

We propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs). While SOTA approaches to Image Classification task work as a black box, there is a growing demand for models that would provide interpreted results. Such a models often learn to predict the distribution over class labels using additional description of this target instances, called concepts. However, existing Bottleneck methods have a number of limitations: their accuracy is lower than that of a standard model and CBMs require an additional set of concepts to leverage. We provide a framework for creating Concept Bottleneck Model from pre-trained multi-modal encoder and new CLIP-like architectures. By introducing a new type of layers known as Concept Bottleneck Layers, we outline three methods for training them: with $\ell_1$-loss, contrastive loss and loss function based on Gumbel-Softmax distribution (Sparse-CBM), while final FC layer is still trained with Cross-Entropy. We show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models. Which means that sparse representation of concepts activation vector is meaningful in Concept Bottleneck Models. Moreover, with our Concept Matrix Search algorithm we can improve CLIP predictions on complex datasets without any additional training or fine-tuning. The code is available at: https://github.com/Andron00e/SparseCBM.

Figures (17)

• Figure 1: Example of concepts extraction with Sparse-CBM.
• Figure 1: Dependence of concept set size on the dataset.
• Figure 2: Overview of our CBM framework.
• Figure 3: Visualization of Sparse-CBM Concept Bottleneck Layers.
• Figure 4: Comparison of our CBM methods on the ImageNet validation subset.