Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Scale Evaluation of Deep Learning-based Explainable Solar Flare Forecasting Models with Attribution-based Proximity Analysis

Temitope Adeyeha, Chetraj Pandey, Berkay Aydin

TL;DR

This study compares two models trained on full-disk line-of-sight (LoS) magnetogram images to predict ≥M-class solar flares within a 24-hour window and introduces a proximity-based metric that quantitatively assesses the accuracy and relevance of local explanations when regions of interest are known.

Abstract

Accurate and reliable predictions of solar flares are essential due to their potentially significant impact on Earth and space-based infrastructure. Although deep learning models have shown notable predictive capabilities in this domain, current evaluations often focus on accuracy while neglecting interpretability and reliability--factors that are especially critical in operational settings. To address this gap, we propose a novel proximity-based framework for analyzing post hoc explanations to assess the interpretability of deep learning models for solar flare prediction. Our study compares two models trained on full-disk line-of-sight (LoS) magnetogram images to predict $\geq$M-class solar flares within a 24-hour window. We employ the Guided Gradient-weighted Class Activation Mapping (Guided Grad-CAM) method to generate attribution maps from these models, which we then analyze to gain insights into their decision-making processes. To support the evaluation of explanations in operational systems, we introduce a proximity-based metric that quantitatively assesses the accuracy and relevance of local explanations when regions of interest are known. Our findings indicate that the models' predictions align with active region characteristics to varying degrees, offering valuable insights into their behavior. This framework enhances the evaluation of model interpretability in solar flare forecasting and supports the development of more transparent and reliable operational systems.

Large Scale Evaluation of Deep Learning-based Explainable Solar Flare Forecasting Models with Attribution-based Proximity Analysis

TL;DR

This study compares two models trained on full-disk line-of-sight (LoS) magnetogram images to predict ≥M-class solar flares within a 24-hour window and introduces a proximity-based metric that quantitatively assesses the accuracy and relevance of local explanations when regions of interest are known.

Abstract

Accurate and reliable predictions of solar flares are essential due to their potentially significant impact on Earth and space-based infrastructure. Although deep learning models have shown notable predictive capabilities in this domain, current evaluations often focus on accuracy while neglecting interpretability and reliability--factors that are especially critical in operational settings. To address this gap, we propose a novel proximity-based framework for analyzing post hoc explanations to assess the interpretability of deep learning models for solar flare prediction. Our study compares two models trained on full-disk line-of-sight (LoS) magnetogram images to predict M-class solar flares within a 24-hour window. We employ the Guided Gradient-weighted Class Activation Mapping (Guided Grad-CAM) method to generate attribution maps from these models, which we then analyze to gain insights into their decision-making processes. To support the evaluation of explanations in operational systems, we introduce a proximity-based metric that quantitatively assesses the accuracy and relevance of local explanations when regions of interest are known. Our findings indicate that the models' predictions align with active region characteristics to varying degrees, offering valuable insights into their behavior. This framework enhances the evaluation of model interpretability in solar flare forecasting and supports the development of more transparent and reliable operational systems.

Paper Structure

This paper contains 9 sections, 5 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Preprocessing Attribution Maps
  4. Explanation Evaluation Metrics
  5. Experimental Evaluation
  6. Discussion
  7. Comparative Analysis of PS Across Models
  8. Comparative Analysis of ACR Across Models
  9. Conclusion

Figures (5)

  • Figure 1: (a) Original input magnetogram image at 2015-01-01T07:59:39.30 UTC. (b) Attribution map generated by Guided Grad-CAM (GGCAM) for input (a). (c) Superimposed image of input (a) and attribution map (b).
  • Figure 2: An schematic overview of steps involved in attribution maps preprocessing and corresponding proximity-based analysis used in this study.
  • Figure 3: An example of sequential processing applied to the attribution maps of an input magnetogram captured on 2015-01-01 at 07:59:39.30 UTC.
  • Figure 4: Distribution of Proximity Scores, grouped in terms of four outcomes of contingency matrix (referred in text as categories). (a) Proximity score distribution for True Positives (TP) and False Positives (FP) (b) Proximity score distribution for False Negatives (FN) and True Negatives (TN).
  • Figure 5: Distribution of Attribution Colocation Ratios for both models, grouped in terms of four outcomes of contingency matrix (referred in text as categories). (a) ACR distribution for True Positives (TP) and False Positives (FP) (b) ACR distribution for False Negatives (FN) and True Negatives (TN).