PRIME: Prioritizing Interpretability in Failure Mode Extraction

TL;DR

This work proposes a novel approach that prioritizes interpretability in this problem: it starts by obtaining human-understandable concepts (tags) of images in the dataset and then analyzes the model's behavior based on the presence or absence of combinations of these tags.

Abstract

In this work, we study the challenge of providing human-understandable descriptions for failure modes in trained image classification models. Existing works address this problem by first identifying clusters (or directions) of incorrectly classified samples in a latent space and then aiming to provide human-understandable text descriptions for them. We observe that in some cases, describing text does not match well with identified failure modes, partially owing to the fact that shared interpretable attributes of failure modes may not be captured using clustering in the feature space. To improve on these shortcomings, we propose a novel approach that prioritizes interpretability in this problem: we start by obtaining human-understandable concepts (tags) of images in the dataset and then analyze the model's behavior based on the presence or absence of combinations of these tags. Our method also ensures that the tags describing a failure mode form a minimal set, avoiding redundant and noisy descriptions. Through several experiments on different datasets, we show that our method successfully identifies failure modes and generates high-quality text descriptions associated with them. These results highlight the importance of prioritizing interpretability in understanding model failures.

Figures (14)

• Figure 1: Visualization of two detected failure modes of class "fox" on a model trained on Living17. Overall accuracy for images of class "fox" is $81.96\%$. However, we identify two coherent subsets of images with significant accuracy drops: foxes standing in dry grass fields ($47.83\%$ accuracy) and foxes in a zoo where a white object (fox or other objects) is detected ($35.29\%$ accuracy). See Appendix \ref{['sec:app-vis']} for more examples.
• Figure 2: PRIME illustration.
• Figure 3: Although appearance of tags "hang", "black", and "branch" individually lowers model's accuracy, when all of them appear in the images, model's accuracy drops from $86.23\%$ to $41.88\%$.
• Figure 4: Accuracy of model over $50$ generated images corresponding to one of the success modes and failure modes for classes "bear", " parrot", and "fox" from Living17. Accuracy gap shows that our method can identify hard and easy subpopulations. Images show that extracted tags are capable of describing detailed images.
• Figure 5: The mean and standard deviation of similarity scores between images in failure modes and their respective descriptions, along with the AUROC measuring the similarity score between descriptions and images inside and outside of failure modes, demonstrate that our method outperforms DOMINO in descriptions it generates for detected failure modes across various datasets.