Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Concept Alignment in Vision-Language Concept Bottleneck Models

Nithish Muthuchamy Selvaraj, Xiaobao Guo, Adams Wai-Kin Kong, Alex Kot

TL;DR

Vision-Language CBMs using expert concepts often exhibit poor faithfulness in concept scores despite strong classification. The authors introduce Contrastive Semi-Supervised (CSS) learning to align concept scores with ground-truth concepts using a small amount of labeled data and a contrastive objective, plus a class-level intervention for confounding in fine-grained tasks. Across CUB, RIVAL, AwA2, and WBCAtt, CSS substantially improves concept accuracy and yields measurable gains in classification with limited supervision, while distributional analyses show denser, more separable concept-space clusters. The work provides a practical path toward trustworthy, interpretable VL-CBMs by combining targeted concept supervision with concept-space interventions.

Abstract

Concept Bottleneck Models (CBM) map images to human-interpretable concepts before making class predictions. Recent approaches automate CBM construction by prompting Large Language Models (LLMs) to generate text concepts and employing Vision Language Models (VLMs) to score these concepts for CBM training. However, it is desired to build CBMs with concepts defined by human experts rather than LLM-generated ones to make them more trustworthy. In this work, we closely examine the faithfulness of VLM concept scores for such expert-defined concepts in domains like fine-grained bird species and animal classification. Our investigations reveal that VLMs like CLIP often struggle to correctly associate a concept with the corresponding visual input, despite achieving a high classification performance. This misalignment renders the resulting models difficult to interpret and less reliable. To address this issue, we propose a novel Contrastive Semi-Supervised (CSS) learning method that leverages a few labeled concept samples to activate truthful visual concepts and improve concept alignment in the CLIP model. Extensive experiments on three benchmark datasets demonstrate that our method significantly enhances both concept (+29.95) and classification (+3.84) accuracies yet requires only a fraction of human-annotated concept labels. To further improve the classification performance, we introduce a class-level intervention procedure for fine-grained classification problems that identifies the confounding classes and intervenes in their concept space to reduce errors.

Improving Concept Alignment in Vision-Language Concept Bottleneck Models

TL;DR

Vision-Language CBMs using expert concepts often exhibit poor faithfulness in concept scores despite strong classification. The authors introduce Contrastive Semi-Supervised (CSS) learning to align concept scores with ground-truth concepts using a small amount of labeled data and a contrastive objective, plus a class-level intervention for confounding in fine-grained tasks. Across CUB, RIVAL, AwA2, and WBCAtt, CSS substantially improves concept accuracy and yields measurable gains in classification with limited supervision, while distributional analyses show denser, more separable concept-space clusters. The work provides a practical path toward trustworthy, interpretable VL-CBMs by combining targeted concept supervision with concept-space interventions.

Abstract

Concept Bottleneck Models (CBM) map images to human-interpretable concepts before making class predictions. Recent approaches automate CBM construction by prompting Large Language Models (LLMs) to generate text concepts and employing Vision Language Models (VLMs) to score these concepts for CBM training. However, it is desired to build CBMs with concepts defined by human experts rather than LLM-generated ones to make them more trustworthy. In this work, we closely examine the faithfulness of VLM concept scores for such expert-defined concepts in domains like fine-grained bird species and animal classification. Our investigations reveal that VLMs like CLIP often struggle to correctly associate a concept with the corresponding visual input, despite achieving a high classification performance. This misalignment renders the resulting models difficult to interpret and less reliable. To address this issue, we propose a novel Contrastive Semi-Supervised (CSS) learning method that leverages a few labeled concept samples to activate truthful visual concepts and improve concept alignment in the CLIP model. Extensive experiments on three benchmark datasets demonstrate that our method significantly enhances both concept (+29.95) and classification (+3.84) accuracies yet requires only a fraction of human-annotated concept labels. To further improve the classification performance, we introduce a class-level intervention procedure for fine-grained classification problems that identifies the confounding classes and intervenes in their concept space to reduce errors.
Paper Structure (14 sections, 2 equations, 4 figures, 6 tables)

This paper contains 14 sections, 2 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Vision-Language Concept Bottleneck Models
  5. Evaluating Faithfulness of VL-CBMs
  6. Enhancing Faithfulness of VL-CBMs
  7. Experiments
  8. Experimental Setting
  9. Improving Concept Alignment
  10. Comparison with other methods
  11. Distributional Analysis of Concept Space
  12. Intervention
  13. Limitations
  14. Conclusion

Figures (4)

  • Figure 1: (a) Modified CLIP (VLM) with concept projection layer which enhances the raw concept scores. (b) Architecture of VL-CBM and an overview of the Contrastive Semi-Supervised (CSS) learning method.
  • Figure 2: Visualization of $top-8$ concept scores for the CUB (birds) and AwA2 (animals) datasets. Incorrectly activated concepts are highlighted in red.
  • Figure 3: Impact of the number of concept labels on CSS VL-CBM and LaBo.
  • Figure 4: t-SNE visualization of CSS (triangles) vs CLIP (circles) concept scores of 5 random classes for CUB test.