LEMON: Local Explanations via Modality-aware OptimizatioN

TL;DR

LEMON tackles the challenge of explainability for multimodal predictors by introducing a model-agnostic local explainer that delivers modality-level and within-modality attributions using a single structured surrogate. It leverages a Sparse Group Lasso surrogate over partitioned, interpretable units to capture both cross-modal contributions and fine-grained feature importances under a fixed query budget. The approach demonstrates competitive deletion-based faithfulness on vision-language VQA with CLIP and LXMERT, and aligns with radiologist annotations in a clinical CaMCheX task, while substantially reducing query costs and runtime. By providing a unified, hierarchy-based view of multimodal evidence, LEMON facilitates auditing and debugging of complex models across domains with practical computational efficiency.

Abstract

Multimodal models are ubiquitous, yet existing explainability methods are often single-modal, architecture-dependent, or too computationally expensive to run at scale. We introduce LEMON (Local Explanations via Modality-aware OptimizatioN), a model-agnostic framework for local explanations of multimodal predictions. LEMON fits a single modality-aware surrogate with group-structured sparsity to produce unified explanations that disentangle modality-level contributions and feature-level attributions. The approach treats the predictor as a black box and is computationally efficient, requiring relatively few forward passes while remaining faithful under repeated perturbations. We evaluate LEMON on vision-language question answering and a clinical prediction task with image, text, and tabular inputs, comparing against representative multimodal baselines. Across backbones, LEMON achieves competitive deletion-based faithfulness while reducing black-box evaluations by 35-67 times and runtime by 2-8 times compared to strong multimodal baselines.

Figures (5)

• Figure 1: Conceptual overview of LEMON. LEMON queries a black-box model with group-structured perturbations and fits a Sparse Group Lasso surrogate to output a two-level explanation: modality contributions and within-modality evidence.
• Figure 2: Overview of the LEMON framework. The pipeline generates model-agnostic explanations for multimodal inputs. The input sample is partitioned into interpretable units to preserve modality structures. By employing a Sparse Group Lasso (SGL) surrogate, LEMON enforces group-wise sparsity to identify key modalities and regions; see Section \ref{['ssec:sgl']} for details. The final outputs include visual heatmaps and text highlights that are evaluated for faithfulness, compactness and cost.
• Figure 3: Qualitative comparison across black-boxes and explainers. Top row (CLIP): CLIP scores the image against the text prompt (question + answer); the higher similarity score indicates strong alignment with the "yes" description. Bottom row (LXMERT): LXMERT takes (question, image) only and predicts "no" (target/GT: "yes"); thus most attributions are negative (blue), indicating evidence against the target. Across both black-boxes, LEMON consistently attributes the decision to the teeth brushing and mouth area, which is semantically plausible for bad breath, while MM-SHAP under-emphasizes the mouth and produces diffuse, coarse evidence and DIME tends to drift towards less relevant hair regions in LXMERT.
• Figure 4: Modality-aware clinical explanation on CaMCheX with REFLACX evidence (Cardiomegaly). The card combines: positive image evidence (red heatmap) overlaid on the frontal CXR with the REFLACX ellipse (green), modality contribution via surrogate weight share (image/text/vitals), and the top contributing text tokens and vitals fields. An additional text-dominant example is provided in Appendix \ref{['app:rq3']}.
• Figure 5: Additional modality-aware explanation (Edema). Same layout as Fig. \ref{['fig:reflacx-examples']}.