SCE-LITE-HQ: Smooth visual counterfactual explanations with generative foundation models

Abstract

Modern neural networks achieve strong performance but remain difficult to interpret in high-dimensional visual domains. Counterfactual explanations (CFEs) provide a principled approach to interpreting black-box predictions by identifying minimal input changes that alter model outputs. However, existing CFE methods often rely on dataset-specific generative models and incur substantial computational cost, limiting their scalability to high-resolution data. We propose SCE-LITE-HQ, a scalable framework for counterfactual generation that leverages pretrained generative foundation models without task-specific retraining. The method operates in the latent space of the generator, incorporates smoothed gradients to improve optimization stability, and applies mask-based diversification to promote realistic and structurally diverse counterfactuals. We evaluate SCE-LITE-HQ on natural and medical datasets using a desiderata-driven evaluation protocol. Results show that SCE-LITE-HQ produces valid, realistic, and diverse counterfactuals competitive with or outperforming existing baselines, while avoiding the overhead of training dedicated generative models.

Figures (7)

• Figure 1: Comparison between SCE-LITE-HQ and traditional counterfactual explanation frameworks. Conventional methods (left) rely on computationally intensive training of dataset-specific generative models, whereas SCE-LITE-HQ (right) leverages pretrained Generative Foundation Models (GFMs). We illustrate counterfactuals generated on the CelebA-HQ dataset for the blond attribute. SCE-LITE-HQ produces counterfactuals at the original image resolution while preserving visual quality. In contrast, FastDIME (left) requires adjusting the image resolution, which leads to lower-quality counterfactuals.
• Figure 2: Overview of the SCE-LITE-HQ algorithm for generating counterfactual images via latent-space optimization guided by a foundation model and classifier gradients.
• Figure 3: Visualization of counterfactual explanations generated at high resolution on the CelebA-HQ dataset using SCE-LITE-HQ and FastDIME as a baseline. SCE-LITE-HQ successfully identifies the confounding attribute of blue eyes, which is highly correlated with blond hair, while FastDIME fails to capture this relationship due to information loss at lower resolutions.
• Figure 4: Comparison of counterfactual explanations generated by different methods across multiple tasks and datasets. For each example, we show the original input, its predicted label, and the counterfactual generated by each method. On the blond hair classification task, SCE-LITE-HQ achieves a favorable balance between Sparsity, realism, and robustness, successfully flipping the model's prediction without excessive perturbations (in contrast to SCE, which overshoots, or other methods, which undershoot). On the Camelyon dataset, SCE-LITE-HQ identifies confounding factors, enabling the generation of counterfactuals that expose spurious correlations; these can be used with the CFKD algorithm to fine-tune the original model and improve accuracy, explaining the substantial performance gains achieved by SCE-LITE-HQ on this dataset.
• Figure 5: High-resolution counterfactual examples on CelebA-HQ. We compare SCE-LITE-HQ against FastDIME, illustrating qualitative differences in visual fidelity and semantic coherence.