Signature Kernel Scoring Rule as Spatio-Temporal Diagnostic for Probabilistic Forecasting

TL;DR

The paper addresses the challenge of evaluating and training probabilistic weather forecasts that exhibit strong spatio-temporal dependencies. It introduces the signature kernel scoring rule, grounded in rough path theory, to encode temporal and spatial structure via path signatures and RKHS kernels. Through WeatherBench 2.0 experiments on ERA5 reanalysis data, it demonstrates that the score is strictly proper and enables effective prequential training of sliding-window generative models, achieving competitive performance and improved discrimination across multi-step forecasts up to about 15 steps. The work provides a practical, theoretically justified framework for forecast verification and training, with clear directions for stability improvements and broader integration into operational forecasting pipelines.

Abstract

Modern weather forecasting has increasingly transitioned from numerical weather prediction (NWP) to data-driven machine learning forecasting techniques. While these new models produce probabilistic forecasts to quantify uncertainty, their training and evaluation may remain hindered by conventional scoring rules, primarily MSE, which ignore the highly correlated data structures present in weather and atmospheric systems. This work introduces the signature kernel scoring rule, grounded in rough path theory, which reframes weather variables as continuous paths to encode temporal and spatial dependencies through iterated integrals. Validated as strictly proper through the use of path augmentations to guarantee uniqueness, the signature kernel provides a theoretically robust metric for forecast verification and model training. Empirical evaluations through weather scorecards on WeatherBench 2 models demonstrate the signature kernel scoring rule's high discriminative power and unique capacity to capture path-dependent interactions. Following previous demonstration of successful adversarial-free probabilistic training, we train sliding window generative neural networks using a predictive-sequential scoring rule on ERA5 reanalysis weather data. Using a lightweight model, we demonstrate that signature kernel based training outperforms climatology for forecast paths of up to fifteen timesteps.

Figures (13)

• Figure 1: Representation of the signed Lévy area of a two-dimensional path. $A_{+}$ and $A_{-}$ denote the positive and negative respective areas between the path (red) and chord (blue). $S^{1,2}$ and $S^{2,1}$ denote the second order terms equal to the area between the path and their perpendicular enclosure (black).
• Figure 2: Deterministic scorecard comparison of Fuxi and IFS Mean across weather variables and metrics, over three regions of the globe. Darker blues signify positive metric performance for Fuxi as compared to the IFS Mean metric value, evaluated across 15 days.
• Figure 3: Probabilistic scorecard comparison of Neural GCM and IFS ENS across weather variables and metrics, over three regions of the globe. Darker blues signify positive metric performance for Neural GCM as compared to the IFS ENS metric value, evaluated across 15 days.
• Figure 4: Diagram of the sliding window prequential generation. The standard one-step prequential generation is applied to each observation window, shifting by one into the newly predicted samples each iteration. The generative net receives designated latent variables $z^e_l$, corresponding to ensemble member $e$ and path $l$.
• Figure 5: Diagram of the implemented U-NET model architecture. The input size is equivalent to the window size of 10. The output size is 1. The U-NET has depths of 32, 64, 128, and 256. The latent variables, introduced between the encoder and decoder, are denoted by Z.