A Multimodal XAI Framework for Trustworthy CNNs and Bias Detection in Deep Representation Learning

TL;DR

The paper tackles trustworthiness and fairness in generative AI by proposing a multimodal XAI framework that integrates attention-based feature fusion, Grad-CAM++ explanations, and a Reveal-to-Revise bias-detection loop. It introduces a latent attribution mechanism, an explainability-constrained optimization to promote disentangled representations, and a human-aligned Cognitive Alignment Score for evaluation. Evaluations on multimodal MNIST demonstrate improvements in predictive performance and explanation fidelity, with ablations highlighting synergy among fusion, interpretability, and bias feedback. The framework offers a practical path toward trustworthy AI in sensitive domains by balancing accuracy, transparency, and fairness in high-stakes pattern recognition tasks.

Abstract

Standard benchmark datasets, such as MNIST, often fail to expose latent biases and multimodal feature complexities, limiting the trustworthiness of deep neural networks in high-stakes applications. We propose a novel multimodal Explainable AI (XAI) framework that unifies attention-augmented feature fusion, Grad-CAM++-based local explanations, and a Reveal-to-Revise feedback loop for bias detection and mitigation. Evaluated on multimodal extensions of MNIST, our approach achieves 93.2% classification accuracy, 91.6% F1-score, and 78.1% explanation fidelity (IoU-XAI), outperforming unimodal and non-explainable baselines. Ablation studies demonstrate that integrating interpretability with bias-aware learning enhances robustness and human alignment. Our work bridges the gap between performance, transparency, and fairness, highlighting a practical pathway for trustworthy AI in sensitive domains.

Figures (10)

• Figure 1: Comparison of true versus predicted values for the black-box model. The red dashed line indicates perfect prediction. The model demonstrates strong alignment with ground truth, achieving a higher $R^2$ and lower RMSE.
• Figure 2: The unified explainable, bias-aware generative framework. The attention-augmented generator $G_\theta$ and Wasserstein critic $D_\phi$ interact in a training loop; bias detection and regularization close the fairness feedback loop, while local explanation and saliency modules provide post-hoc interpretability and privacy-preserving diagnostics.
• Figure 3: CNNs for Bias Detection Grad-CAM Heatmap
• Figure 4: Integration of saliency maps with deep autoencoders enables interpretable bias detection in CNNs, while the gradient-based explanation approach enhances CNN transparency, trustworthiness.
• Figure 5: Model training performance: Training and validation accuracy across epochs, showing smooth convergence toward high performance; and training and validation loss curves demonstrating stable optimization and minimal overfitting, indicating effective learning and convergence.