Language-assisted Vision Model Debugger: A Sample-Free Approach to Finding and Fixing Bugs

TL;DR

A language-assisted diagnostic method that uses texts instead of images to diagnose bugs in vision models based on multi-modal models (eg CLIP) and can identify bugs comprehensible to human experts, uncovering not only known bugs but also previously unknown ones.

Abstract

Vision models with high overall accuracy often exhibit systematic errors in specific scenarios, posing potential serious safety concerns. Diagnosing bugs of vision models is gaining increased attention, however traditional diagnostic approaches require annotation efforts (eg rich metadata accompanying each samples of CelebA). To address this issue,We propose a language-assisted diagnostic method that uses texts instead of images to diagnose bugs in vision models based on multi-modal models (eg CLIP). Our approach connects the embedding space of CLIP with the buggy vision model to be diagnosed; meanwhile, utilizing a shared classifier and the cross-modal transferability of embedding space from CLIP, the text-branch of CLIP become a proxy model to find bugs in the buggy model. The proxy model can classify texts paired with images. During the diagnosis, a Large Language Model (LLM) is employed to obtain task-relevant corpora, and this corpora is used to extract keywords. Descriptions constructed with templates containing these keywords serve as input text to probe errors in the proxy model. Finally, we validate the ability to diagnose existing visual models using language on the Waterbirds and CelebA datasets, we can identify bugs comprehensible to human experts, uncovering not only known bugs but also previously unknown ones.

Figures (6)

• Figure 1: Comparison of different diagnostic solutions. (a) automatically clustering and describing visual failures using image data, (b) using language to prob visual failures on top of a multimodal embedding space, and (c) our method, using language to prob visual failures of arbitrary buggy model.
• Figure 2: The framework of LaVMD. We aim to identify underperforming subgroups of data that exhibit significantly lower performance on a task-irrelevant attribute within a category. In the example above, this refers to the bias subgroups of man and woman with blond hair, where man with blond hair is a specific bias subgroup.
• Figure 3: Discover attributes with high error gaps for each class. The attributes marked in red are the ground truth attributes annotated in three datasets. In NICO++, we only visualize attributes in the class landways.
• Figure 4: Model accuracy on each attribute by retrieving image samples from landbird class in Waterbirds and waterways class in NICO++. We visualize the three attributes with 3 highest $\Delta(y,a)$ and 3 lowest $\Delta(y,a)$.
• Figure 5: Visualize samples corresponding attributes. We choose several understandable attributes predicted by our method for the landways class from NICO++ and present the top-6 samples in these attribute. At the same time, we present accuracy of the top-10 samples .