DEXTER: Diffusion-Guided EXplanations with TExtual Reasoning for Vision Models

TL;DR

DEXTER presents a data-free framework that globally explains visual classifiers by coupling diffusion-based activation maximization with large-language-model reasoning. It optimizes a soft prompt to generate class-consistent images that maximize target neuron activations, then leverages a vision-language model to produce human-readable textual explanations of model decisions and biases without using training data. The approach demonstrates robust capabilities across activation maximization, slice discovery, and bias explanation, validated on SalientImageNet, Waterbirds, CelebA, and FairFaces, with user studies and automated metrics supporting interpretability. This work enables offline, semantically grounded auditing and debiasing of vision models, offering a scalable method to reveal global decision patterns and spurious correlations.

Abstract

Understanding and explaining the behavior of machine learning models is essential for building transparent and trustworthy AI systems. We introduce DEXTER, a data-free framework that employs diffusion models and large language models to generate global, textual explanations of visual classifiers. DEXTER operates by optimizing text prompts to synthesize class-conditional images that strongly activate a target classifier. These synthetic samples are then used to elicit detailed natural language reports that describe class-specific decision patterns and biases. Unlike prior work, DEXTER enables natural language explanation about a classifier's decision process without access to training data or ground-truth labels. We demonstrate DEXTER's flexibility across three tasks-activation maximization, slice discovery and debiasing, and bias explanation-each illustrating its ability to uncover the internal mechanisms of visual classifiers. Quantitative and qualitative evaluations, including a user study, show that DEXTER produces accurate, interpretable outputs. Experiments on ImageNet, Waterbirds, CelebA, and FairFaces confirm that DEXTER outperforms existing approaches in global model explanation and class-level bias reporting. Code is available at https://github.com/perceivelab/dexter.

Figures (13)

• Figure 1: DEXTER investigates classifier biases by optimizing a learnable soft prompt to generate text prompts. These text prompts condition a diffusion model to generate images that maximize the activation of the target class in the vision classifier. Images that correctly activate the target class are stored and later captioned for Bias Reasoning. A VLM reasons using these captions to produce human-understandable textual explanations of the model's decisions and potential biases. More details and clarifications about the pipeline can be found in the Appendices \ref{['app:glossary']} and \ref{['app:algorithm']}.
• Figure 2: User study results on SalientImageNet: Participants evaluated the alignment between classifier attention and visual explanations, categorized into perceptual features (shape, texture, color), conceptual features (context, semantics, multiple elements), and no alignment.
• Figure 3: DEXTER and DiffExplainer explanations for RobustResNet50 on "dog sled", showing spurious top-5 features as core visual elements.
• Figure 4: DEXTER explanation reports.Report 1 analyzes a classifier trained on a FairFaces variant with male overrepresentation in class 1, while Report 2 corresponds to a balanced dataset.
• Figure 5: Comparison of DiffExplainer and DEXTER with respect to SalientImagenet for the neural feature maximization task, covering different classes. The left column displays classes without spurious features, whereas the right column shows classes with spurious features.