Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Verification of the NOAA Space Weather Prediction Center solar flare forecast (1998-2024)

Enrico Camporeale, Thomas E. Berger

Abstract

The NOAA Space Weather Prediction Center (SWPC) issues the official U.S. government forecast for M-class and X-class solar flares, yet the skill of these forecasts has never been comprehensively verified. In this study, we evaluate the SWPC probabilistic flare forecasts over a 26-year period (1998-2024), comparing them to several zero-cost and statistical baselines including persistence, climatology, Naive Bayes, and logistic regression. We find that the SWPC model does not outperform these baselines across key classification and probabilistic metrics and exhibits severe calibration issues and high false alarm rates, especially in high-stakes scenarios such as detecting the first flare after extended quiet periods. These findings demonstrate the need for more accurate and reliable eruption forecasting models which we suggest should be based on modern data-driven methods. The findings also provide a standard against which any proposed eruption prediction system should be compared. We suggest that space weather forecasters regularly update and publish analyses like the one demonstrated here to provide up-to-date standards of accuracy and reliability against which to compare new methods.

Verification of the NOAA Space Weather Prediction Center solar flare forecast (1998-2024)

Abstract

The NOAA Space Weather Prediction Center (SWPC) issues the official U.S. government forecast for M-class and X-class solar flares, yet the skill of these forecasts has never been comprehensively verified. In this study, we evaluate the SWPC probabilistic flare forecasts over a 26-year period (1998-2024), comparing them to several zero-cost and statistical baselines including persistence, climatology, Naive Bayes, and logistic regression. We find that the SWPC model does not outperform these baselines across key classification and probabilistic metrics and exhibits severe calibration issues and high false alarm rates, especially in high-stakes scenarios such as detecting the first flare after extended quiet periods. These findings demonstrate the need for more accurate and reliable eruption forecasting models which we suggest should be based on modern data-driven methods. The findings also provide a standard against which any proposed eruption prediction system should be compared. We suggest that space weather forecasters regularly update and publish analyses like the one demonstrated here to provide up-to-date standards of accuracy and reliability against which to compare new methods.

Paper Structure

This paper contains 23 sections, 1 equation, 6 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Data
  3. Methodology
  4. Confusion Matrix and Binary Metrics
  5. ROC Curve and Optimal Threshold
  6. Probabilistic Forecast Evaluation
  7. Baseline Models and Context for Evaluation Metrics
  8. Persistence Model.
  9. Climatology-Based and Statistical Models.
  10. Training Strategy
  11. Thresholding for Binary Metrics.
  12. Interpreting Metrics in Imbalanced Contexts
  13. Results
  14. Reliability of SWPC Probabilistic Forecasts
  15. Standard threshold: $\theta=0.5$
  16. ...and 8 more sections

Figures (6)

  • Figure 1: Number of positive days (M-class or X-class) in a 27-day sliding window, plotted over the study period. Blue dots indicate the 27-days average of sunspot numbers. Modulation due to the approximately 11-year solar magnetic activity cycle is clearly visible.
  • Figure 2: Total number of M-class and X-class flares as a function of day of year, aggregated over the entire dataset.
  • Figure 3: Conditional probability of observing an M-class or X-class flare given $n$ consecutive prior days without such a flare.
  • Figure 4: Reliability diagrams for M-class and X-class flare forecasts at 24-hour, 48-hour, and 72-hour lead times. The diagonal dashed line represents perfect reliability.
  • Figure 5: Confusion matrix for 'storm after the calm' scenario (X flares)
  • ...and 1 more figures