A novel neural network-based approach to derive a geomagnetic baseline for robust characterization of geomagnetic indices at mid-latitude

Abstract

Geomagnetic indices derived from ground magnetic measurements characterize the intensity of solar-terrestrial interaction. The \textit{Kp} index derived from multiple magnetic observatories at mid-latitude has commonly been used for space weather operations. Yet, its temporal cadence is low and its intensity scale is crude. To derive a new generation of geomagnetic indices, it is desirable to establish a geomagnetic `baseline' that defines the quiet-level of activity without solar-driven perturbations. We present a new approach for deriving a baseline that represents the time-dependent quiet variations focusing on data from Chambon-la-Forêt, France. Using a filtering technique, the measurements are first decomposed into the above-diurnal variation and the sum of 24h, 12h, 8h, and 6h filters, called the daily variation. Using correlation tools and SHapley Additive exPlanations, we identify parameters that dominantly correlate with the daily variation. Here, we predict the daily `quiet' variation using a long short-term memory neural network trained using at least 11 years of data at 1h cadence. This predicted daily quiet variation is combined with linear extrapolation of the secular trend associated with the intrinsic geomagnetic variability, which dominates the above-diurnal variation, to yield a new geomagnetic baseline. Unlike the existing baselines, our baseline is insensitive to geomagnetic storms. It is thus suitable for defining geomagnetic indices that accurately reflect the intensity of solar-driven perturbations. Our methodology is quick to implement and scalable, making it suitable for real-time operation. Strategies for operational forecasting of our geomagnetic baseline 1 day and 27 days in advance are presented.

Figures (10)

• Figure 1: (a) Linear correlation coefficients between parameters and (b) mutual information between the dependent parameters (vertical) and independent parameters (horizontal), using data between 1997 and 2007 (11 years in total).
• Figure 2: (a) Comparison between $Y_D$ (black) and the modeled $Y_D$ (red) using XGBoost for the interval between July 11 and 19, 2012, including an ICME passage between July 14 at 18:09 and July 17 at 05:00 as shaded in grey. (b) Contribution of the input features to SHAP values for the modeled $Y_D$. (c) Contribution of the four most-important features to SHAP values for the modeled $Y_D$. The ICME passage is delineated by grey dotted lines in panels (b) and (c).
• Figure 3: (a) $F_{10.7}$, SZA, DistSE, and LT during June 1 and 5, 2009, highlighted in green as an example for the sequential inputs to the neural network. (b) $X_D$, $Y_D$, and $Z_D$ for the same interval, marked with red dots for the expected prediction. (c) Schematic of the neural network consisting of stacked layers and multiple RNN units (nodes), taken here as LSTM cells. The neural network takes in 12-hour sequences (green) of LT, SZA, DistSE, and $F_{10.7}$ up to the current time stamp $t_N$ and predicts the daily quiet variations (red) at the next adjacent hour $t_{N+1}$.
• Figure 4: The walk forward training approach. The $F_{10.7}$ (black) indicative of the solar variability is shown for context. (a) Step 1 consists in training of the neural network with a specified training window (blue shade) and validating it with the adjacent data with a specified validation window (green shade). (b) Step 2 consists in updating the training with the next, shifted training and validation windows. The process is repeated up until to the test year. (c) The model is tested after the final training and validation step.
• Figure 5: Comparison between the daily filter (black) and the neural network (NN) prediction of the daily quiet variation (red) during the consecutive CK48 days between May 18 and June 6, 2009, shown for (a) $X_D$, (b) $Y_D$, and (c) $Z_D$ components. Passages of CIRs on May 20 - 21 and May 28 are shaded in grey.