CAuSE: Decoding Multimodal Classifiers using Faithful Natural Language Explanation

TL;DR

<CAuSE> addresses the opacity of multimodal classifiers by grounding post-hoc natural language explanations in the classifier's internal reasoning through causal abstraction via Interchange Intervention Training. The framework trains a specialized Explainer (ψ, φ, 𝒜, 𝒞2) to simulate the base classifier and become a causal abstraction, guided by L_φ, L_TS, L_IIT, and R_match losses. A novel CCMR metric evaluates faithfulness in multimodal settings by measuring counterfactual consistency in representation space. Empirical results across e-SNLI-VE, Hateful Memes, and VQA-X show CAuSE achieves strong faithfulness and competitive plausibility, with thorough qualitative analyses and error analyses supporting its strengths and limitations. The work provides a scalable blueprint for obtaining faithful, task- and architecture-agnostic explanations for discriminative multimodal systems.>

Abstract

Multimodal classifiers function as opaque black box models. While several techniques exist to interpret their predictions, very few of them are as intuitive and accessible as natural language explanations (NLEs). To build trust, such explanations must faithfully capture the classifier's internal decision making behavior, a property known as faithfulness. In this paper, we propose CAuSE (Causal Abstraction under Simulated Explanations), a novel framework to generate faithful NLEs for any pretrained multimodal classifier. We demonstrate that CAuSE generalizes across datasets and models through extensive empirical evaluations. Theoretically, we show that CAuSE, trained via interchange intervention, forms a causal abstraction of the underlying classifier. We further validate this through a redesigned metric for measuring causal faithfulness in multimodal settings. CAuSE surpasses other methods on this metric, with qualitative analysis reinforcing its advantages. We perform detailed error analysis to pinpoint the failure cases of CAuSE. For replicability, we make the codes available at https://github.com/newcodevelop/CAuSE

Figures (10)

• Figure 1: This figure shows the abstract schematic of CAuSE and how it explains discriminative multimodal classifiers at inference. The internal components of CAuSE are i) an MLP $\psi$, ii) A language model (LM) $\phi$, and iii) an aggregator followed by a classifier $\mathcal{C}_2$. The input to the multimodal encoder is a meme which is composed of both text and image (separately not shown).
• Figure 2: This toy diagram shows the training process of the CAuSE framework, specifically the calculations of $\mathcal{L}_{TS}$ and $\mathcal{L}_{IIT}$ losses on a sample from e-SNLI-VE dataset. The input vectors to $\mathcal{C}_1$ and $\mathcal{C}_2$ are arbitrary but depicts a real scenario.
• Figure 3: Examples illustrating the faithfulness–plausibility trade-off across the three datasets. In each case, $\phi$ produces a more plausible explanation for an incorrect prediction (i.e. unfaithful), while CAuSE generates a less plausible but more faithful (reflecting a correct prediction) explanation. Both $\phi$ and CAuSE seek to emulate the classifier’s output and provide justifications consistent with the predicted answer. GL: Ground-truth label, M: Prediction from $M$.
• Figure 4: Representative dataset examples where CAuSE outperforms ablations in the LLM tournament.
• Figure 5: Representative dataset examples where CAuSE underperforms ablations in fluent and coherent generation.

Theorems & Definitions (5)

• proof
• Lemma 1
• proof
• Theorem 1
• proof