Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

No Token Left Behind: Explainability-Aided Image Classification and Generation

Roni Paiss, Hila Chefer, Lior Wolf

TL;DR

The paper tackles instability in zero-shot CLIP by identifying neglect of semantic tokens during similarity optimization.It introduces an explainability-based loss that leverages Transformer explainability maps to enforce token-level and spatial attention, applied to prompt engineering and CLIP-guided generation.Across one-shot classification, zero-shot image manipulation, and spatially conditioned text-to-image generation, the method yields more faithful and semantically faithful results than strong baselines, with robust performance across backbones and tasks.The work demonstrates that explainability can serve as a practical feedback signal to enhance multimodal, zero-shot systems, though it acknowledges remaining gaps relative to supervised approaches.

Abstract

The application of zero-shot learning in computer vision has been revolutionized by the use of image-text matching models. The most notable example, CLIP, has been widely used for both zero-shot classification and guiding generative models with a text prompt. However, the zero-shot use of CLIP is unstable with respect to the phrasing of the input text, making it necessary to carefully engineer the prompts used. We find that this instability stems from a selective similarity score, which is based only on a subset of the semantically meaningful input tokens. To mitigate it, we present a novel explainability-based approach, which adds a loss term to ensure that CLIP focuses on all relevant semantic parts of the input, in addition to employing the CLIP similarity loss used in previous works. When applied to one-shot classification through prompt engineering, our method yields an improvement in the recognition rate, without additional training or fine-tuning. Additionally, we show that CLIP guidance of generative models using our method significantly improves the generated images. Finally, we demonstrate a novel use of CLIP guidance for text-based image generation with spatial conditioning on object location, by requiring the image explainability heatmap for each object to be confined to a pre-determined bounding box.

No Token Left Behind: Explainability-Aided Image Classification and Generation

TL;DR

The paper tackles instability in zero-shot CLIP by identifying neglect of semantic tokens during similarity optimization.It introduces an explainability-based loss that leverages Transformer explainability maps to enforce token-level and spatial attention, applied to prompt engineering and CLIP-guided generation.Across one-shot classification, zero-shot image manipulation, and spatially conditioned text-to-image generation, the method yields more faithful and semantically faithful results than strong baselines, with robust performance across backbones and tasks.The work demonstrates that explainability can serve as a practical feedback signal to enhance multimodal, zero-shot systems, though it acknowledges remaining gaps relative to supervised approaches.

Abstract

The application of zero-shot learning in computer vision has been revolutionized by the use of image-text matching models. The most notable example, CLIP, has been widely used for both zero-shot classification and guiding generative models with a text prompt. However, the zero-shot use of CLIP is unstable with respect to the phrasing of the input text, making it necessary to carefully engineer the prompts used. We find that this instability stems from a selective similarity score, which is based only on a subset of the semantically meaningful input tokens. To mitigate it, we present a novel explainability-based approach, which adds a loss term to ensure that CLIP focuses on all relevant semantic parts of the input, in addition to employing the CLIP similarity loss used in previous works. When applied to one-shot classification through prompt engineering, our method yields an improvement in the recognition rate, without additional training or fine-tuning. Additionally, we show that CLIP guidance of generative models using our method significantly improves the generated images. Finally, we demonstrate a novel use of CLIP guidance for text-based image generation with spatial conditioning on object location, by requiring the image explainability heatmap for each object to be confined to a pre-determined bounding box.
Paper Structure (25 sections, 10 equations, 15 figures, 11 tables, 1 algorithm)

This paper contains 25 sections, 10 equations, 15 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Explainability
  5. Prompt engineering
  6. Zero-shot text-guided image manipulation
  7. Zero-shot text-to-image with spatial conditioning
  8. Experiments
  9. One-shot prompt engineering
  10. Zero-shot text-guided image manipulation
  11. Zero-shot text-to-image with spatial conditioning
  12. Discussion
  13. Conclusions
  14. Acknowledgments
  15. Explainability Method
  16. ...and 10 more sections

Figures (15)

  • Figure 1: Manipulations for "A person with purple hair". StyleClip patashnik2021styleclip produces a manipulation that is not consistent with the semantic meaning of the prompt, and the color of the person's shirt and the background are altered. Our method generates an output that is faithful to the input text query, and the high values of the explainability heatmaps are much more correlated with the prompt.
  • Figure 2: A qualitative comparison of prompt engineering using CoOp zhou2021coop with and without our method on 2 exemplary samples from ImageNetV2 pmlr-v97-recht19a. We present the relevance maps for the ground truth class chosen by our method ("necklace", "jigsaw"), and the counterfactual class chosen by CoOp ("bolo tie", "maraca"). The learned vectors for the prompt are annotated by the letter "v" in the textual explainability maps, since the vectors do not represent actual tokens. As can be seen, for the ground truth classes "necklace" and "jigsaw", our prompts encourage CLIP to focus on the class name in the input text, while CoOp leads CLIP to consider unrelated tokens. This can cause CLIP to produce biased similarity scores based on the engineered prompts.
  • Figure 3: A qualitative comparison between StyleCLIP (SC) and our method on 4 different textual prompts. (a) "A man with a beard", (b) "A person with purple hair", (c) "A blond man", (d) "A person with grey hair". For each prompt we present examples where StyleCLIP is successful (right column), and unsuccessful (left column). For the failure cases, the optimization in StyleCLIP hardly modifies the original image, leading to a high identity preservation score when no semantic change was applied. When StyleCLIP is successful, our method produces similar or identical results.
  • Figure 4: A qualitative comparison between the two similarity-based baselines and our method for CLIP-guided zero-shot text-based image generation with spatial conditioning. Textual conditioning refers to specifying the spatial positioning of objects within the text prompts, for example "a vase on a table". Additional examples are presented in App. \ref{['app:l2i_examples']}.
  • Figure 5: 1-shot accuracy (in percentage) on the ImageNet test set for different choices of $\lambda_{expl}$ for all visual backbones of CLIP. The accuracy achieved by the baselines is denoted as $\lambda_{expl}=0$.
  • ...and 10 more figures