Zero-shot Concept Bottleneck Models

TL;DR

The paper introduces Zero-shot Concept Bottleneck Models (Z-CBMs), which perform fully zero-shot input-to-concept and concept-to-label inference by retrieving concepts from a massive concept bank and using sparse linear regression to select essential concepts for label prediction, all without task-specific training. By leveraging pre-trained vision-language models and a scalable cross-modal retrieval pipeline (Faiss) with $K$-best concept candidates, Z-CBMs reconstruct input features through a weighted concept vector and classify in a zero-shot fashion, providing interpretable and intervenable explanations. Across 12 diverse datasets, Z-CBMs achieve strong CLIP-Score-based concept explanations and competitive zero-shot classification performance, with improvements driven by larger concept banks, backbone VLM strength, and sparse concept regressors. The work demonstrates practical zero-shot interpretability for multi-domain inputs and establishes a flexible baseline that can adapt to evolving VLMs while enabling human intervention and analysis.

Abstract

Concept bottleneck models (CBMs) are inherently interpretable and intervenable neural network models, which explain their final label prediction by the intermediate prediction of high-level semantic concepts. However, they require target task training to learn input-to-concept and concept-to-label mappings, incurring target dataset collections and training resources. In this paper, we present \textit{zero-shot concept bottleneck models} (Z-CBMs), which predict concepts and labels in a fully zero-shot manner without training neural networks. Z-CBMs utilize a large-scale concept bank, which is composed of millions of vocabulary extracted from the web, to describe arbitrary input in various domains. For the input-to-concept mapping, we introduce concept retrieval, which dynamically finds input-related concepts by the cross-modal search on the concept bank. In the concept-to-label inference, we apply concept regression to select essential concepts from the retrieved concepts by sparse linear regression. Through extensive experiments, we confirm that our Z-CBMs provide interpretable and intervenable concepts without any additional training. Code will be available at https://github.com/yshinya6/zcbm.

Figures (10)

• Figure 1: Zero-shot concept bottleneck models (Z-CBMs). Z-CBMs predict concepts for input by retrieving them from a large-scale concept bank. Then, Z-CBMs predict labels based on the weighted sum of the retrieved concept vectors with importance weights yielded by sparse linear regression.
• Figure 2: Concept Retrieval
• Figure 3: Concept Regression
• Figure 5: Concept Deletion (Bird)
• Figure 6: Concept Insertion (Bird)