Explainability for Vision Foundation Models: A Survey

TL;DR

This survey interrogates explainability in vision foundation models (PFMs), highlighting how PFMs both pose interpretability challenges and enable new explainable AI designs. It categorizes XAI methods into inherently explainable approaches (e.g., CBMs, textual rationales, CoT, prototypical networks) and post-hoc techniques, detailing how PFMs are used across each category and their evaluation. It analyzes axioms and metrics for assessing explanations, notes the dominance of text-image multimodality, and discusses current challenges (bias, reasoning, background knowledge) along with open questions (mathematical grounding, new modalities, quantitative evaluation, spurious explanations). The paper argues for future work that models latent spaces and outputs with stronger theoretical grounding while expanding modalities and establishing standardized evaluation benchmarks to reliably compare PFMs-based XAI methods. The findings have practical significance for deploying trustworthy, interpretable vision systems that leverage large, multimodal models in real-world settings.

Abstract

As artificial intelligence systems become increasingly integrated into daily life, the field of explainability has gained significant attention. This trend is particularly driven by the complexity of modern AI models and their decision-making processes. The advent of foundation models, characterized by their extensive generalization capabilities and emergent uses, has further complicated this landscape. Foundation models occupy an ambiguous position in the explainability domain: their complexity makes them inherently challenging to interpret, yet they are increasingly leveraged as tools to construct explainable models. In this survey, we explore the intersection of foundation models and eXplainable AI (XAI) in the vision domain. We begin by compiling a comprehensive corpus of papers that bridge these fields. Next, we categorize these works based on their architectural characteristics. We then discuss the challenges faced by current research in integrating XAI within foundation models. Furthermore, we review common evaluation methodologies for these combined approaches. Finally, we present key observations and insights from our survey, offering directions for future research in this rapidly evolving field.

Figures (13)

• Figure 1: Global goal of XAI. While a non explainable method only makes inference, an explainable model produces details about the reasons of its decisions, to make its functioning clear or easy to understand
• Figure 2: Chronology of the order of magnitude of the number of parameters of learning methods. While early methods were interpretable and lightweight, subsequent developments have led to an increase in complexity that has culminated in foundation models, which are mainly characterized by their size.
• Figure 3: Chain representing the acceptance of AI in society. Each box presents the involved audience (middle) and the description of the step (bottom).
• Figure 4: Summary of the XAI methods presented in our study.
• Figure 5: Scheme of the principle of an inherently interpretable model through a concept bottleneck. Given an input, the CBM first generates a conceptual representation based on a predefined set of concepts. Subsequently, the model produces an output using this conceptual representation.