Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioning

TL;DR

<3-5 sentence high-level summary> This paper presents a proof-of-concept pipeline that uses off-the-shelf text-to-image and image-to-text models to automatically discover and describe failures of vision models in open-ended, real-world-like settings. By generating synthetic inputs conditioned on true labels, clustering misclassified inputs, and grounding failure explanations in natural language, the approach reveals spurious correlations and enables counterfactual analysis. The method is validated on ImageNet classifiers (including ResNet-50 and ViT variants) and scaled to produce large adversarial datasets with cross-architecture generalization, even extending to real images via Google Image Search. The work discusses limitations such as coverage, bias, captioning reliability, and distribution alignment, highlighting both the potential and the caveats of using large generative models for debugging vision systems.

Abstract

Automatically discovering failures in vision models under real-world settings remains an open challenge. This work demonstrates how off-the-shelf, large-scale, image-to-text and text-to-image models, trained on vast amounts of data, can be leveraged to automatically find such failures. In essence, a conditional text-to-image generative model is used to generate large amounts of synthetic, yet realistic, inputs given a ground-truth label. Misclassified inputs are clustered and a captioning model is used to describe each cluster. Each cluster's description is used in turn to generate more inputs and assess whether specific clusters induce more failures than expected. We use this pipeline to demonstrate that we can effectively interrogate classifiers trained on ImageNet to find specific failure cases and discover spurious correlations. We also show that we can scale the approach to generate adversarial datasets targeting specific classifier architectures. This work serves as a proof-of-concept demonstrating the utility of large-scale generative models to automatically discover bugs in vision models in an open-ended manner. We also describe a number of limitations and pitfalls related to this approach.

Figures (16)

• Figure 1: Examples of automatically found failures. The correct label is to the left in green. The incorrect prediction is to the right in red. Images are not watermarked upon classification.
• Figure 2: Diagram of our method. The method starts by generating images containing a given class $y$ to measure the baseline failure rate of that class (right-hand side of \ref{['eq:1']}). We construct a textual description for each misclassified image. This description is used to produce new images and measure the failure rate on images corresponding to that description (left-hand side of \ref{['eq:1']}). The final description can be edited (manually or automatically) to understand the source of the failures.
• Figure 3: Distribution of failures of a ResNet-50 for the baseline and automatically discovered captions for three true and target label pairs. For each panel, failures resulting from the baseline caption are on the left and failures resulting from the discovered caption are on the right. We show the top-15 mistakes and three randomly sampled images for each caption. We highlight in red the bar corresponding to the target label. Absolute failure rates are given in \ref{['table:openended_failures']}.
• Figure 4: Failure rates (top-3) for different models on two generated datasets and ImageNet. We report the failure rates of different models trained on ImageNet.
• Figure 5: Illustrative examples of various challenges.\ref{['fig:limitations_a']} Persian cats in snow (generated using "a realistic photograph of a Persian cat (domestic animal). it is walking in the snow.") are misclassified as snow leopards at a rate of 0.016%, which is significantly higher than the failure rate of 0.0032% induced by the automatically found caption ("--- " --- the background is green."); the total failure rate also increases twelve-fold to 8.15% (from 0.64%). \ref{['fig:limitations_b']} It is estimated that only 1 in 10,000 crayfish turn blue. However, 9% of the images generated using "a realistic photograph of a crayfish (crustacean)." contain a blue crayfish (estimated by manually looking at 100 samples). \ref{['fig:limitations_c']} This image of a crayfish is misclassified as a chainlink fence. The output of the captioning model for this particular image is "a realistic photograph of a crayfish. the crayfish is very detailed. the crayfish is facing the camera. the crayfish is orange. it has two antennae." While the caption describes the image, it does not provide enough detail to reconstruct the image. \ref{['fig:limitations_d']} This image is generated from the caption "a realistic photograph of a saltshaker (container). there is a lemon slice on the side of the salt shaker." While the image contains a lemon, the true class $y$ (saltshaker) is not visible. \ref{['fig:limitations_e']} Generated with the caption "a realistic photograph of a ladybug (insect). it is in a plastic bag.", this image illustrates that text-to-image models can create image that are not from the intended distribution (i.e., of realistic photographs).