Graph Integrated Multimodal Concept Bottleneck Model

TL;DR

MoE-SGT advances Concept Bottleneck Models by embedding a structure-aware Graph Transformer over two heterogeneous graphs (answer–concept and answer–question) and by introducing a Mixture-of-Experts module to adapt model capacity to input complexity. The framework leverages an LLM-driven concept pool, human interventions, and multilingual visual-text priors to build a robust multimodal concept bottleneck vector that feeds a graph-based classifier. Across single- and multi-label datasets, MoE-SGT achieves state-of-the-art or competitive performance, with ablations highlighting the critical role of QA information, graph-based reasoning, and expert routing. This approach offers interpretable concept-based reasoning with scalable capacity, suitable for safety-critical and cross-modal applications, while outlining paths for domain generalization and real-time concept adaptation.

Abstract

With growing demand for interpretability in deep learning, especially in high stakes domains, Concept Bottleneck Models (CBMs) address this by inserting human understandable concepts into the prediction pipeline, but they are generally single modal and ignore structured concept relationships. To overcome these limitations, we present MoE-SGT, a reasoning driven framework that augments CBMs with a structure injecting Graph Transformer and a Mixture of Experts (MoE) module. We construct answer-concept and answer-question graphs for multimodal inputs to explicitly model the structured relationships among concepts. Subsequently, we integrate Graph Transformer to capture multi level dependencies, addressing the limitations of traditional Concept Bottleneck Models in modeling concept interactions. However, it still encounters bottlenecks in adapting to complex concept patterns. Therefore, we replace the feed forward layers with a Mixture of Experts (MoE) module, enabling the model to have greater capacity in learning diverse concept relationships while dynamically allocating reasoning tasks to different sub experts, thereby significantly enhancing the model's adaptability to complex concept reasoning. MoE-SGT achieves higher accuracy than other concept bottleneck networks on multiple datasets by modeling structured relationships among concepts and utilizing a dynamic expert selection mechanism.

Figures (5)

• Figure 1: Overview of the framework. We extract candidate concepts with an LLM to form the final concept pool. Then we generate predicted and prior concept scores for each image with optional intervention, and optimize with alignment and sparsity losses to obtain the multimodal concept bottleneck vector. We generate questions and answers based on the concepts, extract features with MedCLIP, construct answer–concept and answer–question heterogeneous graphs, and then extract graph features through multiple MoE-SGT modules. Finally, these features are fed into an MLP classifier for disease classification.
• Figure 2: Example of concept‐score clamping during the training intervention phase: after fusing multi‐view priors, explicitly present (or absent) concepts are clamped to 1 or 0.
• Figure 3: Example of normalized concept score outputs on the CUB dataset for our model, Sparse‐CBMsparseCBM, VLG‐CBMvlg-cbm, and Label‐Free CBMlabelCBM. Concepts whose error relative to the ground truth is within 0.3 are highlighted in green (accurate), those with error between 0.3 and 0.5 in yellow (inaccurate), and those with error above 0.5 in red (wrong).
• Figure 4: Single-label Top-1 accuracy regarding the number of MoE experts on CUB-200, ImageNet, CIFAR-10, and CIFAR-100.
• Figure 5: Multi‐label performance (ROC‐AUC and F1) regarding the number of MoE experts on MIMIC‐CXR and CheXpert.