FastRM: An efficient and automatic explainability framework for multimodal generative models

TL;DR

FastRM tackles the efficiency bottleneck of explainability in multimodal LVLMs by introducing a gradient-free proxy that predicts relevancy maps from the model's last hidden states. It combines a lightweight normalization-plus-single-head-attention module with an entropy-based uncertainty measure to enable on-the-fly explanations and reliability assessment, reporting $99.8\%$ time savings and $44\%$ memory reductions relative to gradient-based baselines. The approach generalizes across VQA, GQA, and POPE datasets, demonstrating practical potential for real-world deployment in trustworthy AI. By enabling fast, interpretable validations of model outputs and quantifying confidence, FastRM advances scalable grounding of multimodal reasoning in LVLMs.

Abstract

Large Vision Language Models (LVLMs) have demonstrated remarkable reasoning capabilities over textual and visual inputs. However, these models remain prone to generating misinformation. Identifying and mitigating ungrounded responses is crucial for developing trustworthy AI. Traditional explainability methods such as gradient-based relevancy maps, offer insight into the decision process of models, but are often computationally expensive and unsuitable for real-time output validation. In this work, we introduce FastRM, an efficient method for predicting explainable Relevancy Maps of LVLMs. Furthermore, FastRM provides both quantitative and qualitative assessment of model confidence. Experimental results demonstrate that FastRM achieves a 99.8% reduction in computation time and a 44.4% reduction in memory footprint compared to traditional relevancy map generation. FastRM allows explainable AI to be more practical and scalable, thereby promoting its deployment in real-world applications and enabling users to more effectively evaluate the reliability of model outputs.

Figures (10)

• Figure 1: Overview of FastRM. Given an input, the LVLM produces hidden states for each generated token. Then, FastRM generates the relevancy map $R_{FastRM}$ which is subsequently compared to $R_{GT}$ obtained after binarizing the baseline relevancy map.
• Figure 2: Left: Distribution of image patches relevancy scores for the baseline and FastRM-7. Showing patches with relevancy scores exceeding 5% of the maximum relevancy value for each image, while the inset shows the full distribution. Right: shows the distribution of the output lengths.
• Figure 3: Performance of LLaVA v1.5-7B: Accuracy and F1 score of FastRM-7 and FastRM-13 across different classification thresholds.
• Figure 4: Perturbation-based evaluation. For positive perturbation, PP (smaller AUC is better). For negative perturbation, NP (larger AUC is better).
• Figure 5: Ablations studies for FastRM-13. The figure shows how the labeling threshold, the training data size and the number of training steps affect the PP-based evaluation. Smaller AUC is better