Achieving Data Efficient Neural Networks with Hybrid Concept-based Models

TL;DR

This paper addresses data efficiency in supervised learning by proposing hybrid concept-based models that leverage both concept predictions and information outside the concepts to improve downstream task accuracy, especially in sparse data settings. It introduces two architectures (CBM-Res and CBM-Skip) and a Sequential Concept Model (SCM), along with ConceptShapes, a flexible synthetic benchmark for concept-based methods, and an adversarial concept-attack framework to probe robustness. Empirical results show that hybrid models outperform standard CNNs and vanilla concept-based models under limited data, while ConceptShapes demonstrate reliable concept learning, unlike CUB where concepts are often ambiguous. The work also exposes vulnerabilities to adversarial concept attacks, which question the promised interpretability of concept-based approaches and motivate stronger benchmarks and defenses for future research.

Abstract

Most datasets used for supervised machine learning consist of a single label per data point. However, in cases where more information than just the class label is available, would it be possible to train models more efficiently? We introduce two novel model architectures, which we call hybrid concept-based models, that train using both class labels and additional information in the dataset referred to as concepts. In order to thoroughly assess their performance, we introduce ConceptShapes, an open and flexible class of datasets with concept labels. We show that the hybrid concept-based models outperform standard computer vision models and previously proposed concept-based models with respect to accuracy, especially in sparse data settings. We also introduce an algorithm for performing adversarial concept attacks, where an image is perturbed in a way that does not change a concept-based model's concept predictions, but changes the class prediction. The existence of such adversarial examples raises questions about the interpretable qualities promised by concept-based models.

Figures (12)

• Figure 1: Adversarial Concept Attack. Images are perturbed in a way that does not change a concept-based model's concept predictions, but change the class prediction. This brings into question the interpretable qualities of these models.
• Figure 2: Architecture of a CBM-Res used for computer vision. The model uses both concept predictions and a skip connection that hops over the bottleneck layer to perform the output prediction. The architecture can be adapted to a CBM-Skip by performing concatenation instead of addition before the output layer.
• Figure 3: Architecture of a Sequential Bottleneck Model (SCM). The concepts are predicted sequentially throughout the layers, and concatenated together with the final hidden layer before the output layer.
• Figure 4: Images from different classes of two ConceptShapes datasets. Left: Nine different images from a 10-class 5-concept dataset. Right: Nine different images from a 21-class 9-concept dataset.
• Figure 5: Test-set accuracies on the CUB dataset. The x-axis indicates the average amount of images included in the training and validation dataset for each class, where the rightmost point corresponds to the full dataset. The results are averaged over three runs, and include 95% confidence intervals. The oracle model consistently got 100% accuracy and is omitted.