Part-based Quantitative Analysis for Heatmaps

TL;DR

This work introduces Part-based Quantitative Analysis for Heatmaps (PQAH), a quantitative framework that measures heatmap activation overlap with semantic object parts to produce fine-grained, objective XAI insights. By leveraging part annotations and a defined overlap metric (PH) based on $F_1$, PQAH generates per-part and background scores, aggregates them into quartiles $PH_{Q1}$, $PH_{Q2}$, and $PH_{Q3}$, and visualizes the results with boxplots. The authors couple PQAH with a pipeline for automated, end-user XAI reporting via large language models (e.g., GPT-4) and demonstrate the approach on PartImageNet and PASCAL-Part using multiple backbones (ResNet-50, VGG-16, ViT). They further illustrate a medical use case where PQAH-guided data augmentation reduces region-based biases and improves diagnostic accuracy, underscoring PQAH’s utility for both model debugging and user-friendly explanations. The work highlights that PQAH provides a scalable, granular lens for heatmap evaluation and model improvement, while acknowledging that human-centered interpretation remains a consideration and suggesting directions like specialized LLMs to enhance critique generation.

Abstract

Heatmaps have been instrumental in helping understand deep network decisions, and are a common approach for Explainable AI (XAI). While significant progress has been made in enhancing the informativeness and accessibility of heatmaps, heatmap analysis is typically very subjective and limited to domain experts. As such, developing automatic, scalable, and numerical analysis methods to make heatmap-based XAI more objective, end-user friendly, and cost-effective is vital. In addition, there is a need for comprehensive evaluation metrics to assess heatmap quality at a granular level.

Figures (23)

• Figure 1: An overview of the current landscape of heatmap-based XAI and heatmap evaluation methods. Conventional qualitative methods for explaining AI models through heatmaps (a), and evaluating the quality of these heatmaps (c), have predominantly concentrated on visualizing heatmaps for a limited number of examples. In contrast, a quantitative method for explaining AI model performance using heatmaps is currently lacking (b), and the existing quantitative methods for heatmap evaluation (d) rely on simplistic summary statistics which fail to consider detailed fine-grained information.
• Figure 2: Overview of the PQAH (Part-based Quantitative Analysis for Heatmaps) framework. The process involves (1) extracting part masks and heatmaps from the given image dataset, (2) computing the PQAH Overlap scores for each semantic part of the main object in the images, and (3) aggregating $\mathit{PH}$ scores across all semantic part categories to generate statistical summaries and visual representations.
• Figure 3: Exp. 1: Representative examples of PQAH analysis for DNN models. On the X-axis, various parts are displayed, with 'Bg' denoting the background. Complete PQAH analysis can be found in the supplementary materials.
• Figure 4: Exp. 1: Example heatmap visualizations. The visualization method is GradCam + SESS.
• Figure 5: Representative examples of PQAH analysis of Exp. 2-4. The backbone network is ResNet-50. On the X-axis, various parts are displayed, with 'Bg' denoting the background.