Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Tolerance-Based Framework for Spatio-Temporal Forecast Validation Using the gamma-Index

Cyril Voyant

TL;DR

The paper addresses the problem of over-penalizing small spatial or temporal displacements in gridded forecast verification by introducing a three-dimensional $\gamma$-Index that jointly incorporates spatial (DTA), temporal (TTA), and intensity (IDT) tolerances. The method defines a normalized acceptance criterion, $\gamma(x,y,t)$, as a minimum over spatial and temporal neighborhoods, and passes forecasts when $\gamma \le 1$, yielding metrics such as the passing rate $GPR$, mean $\bar{\gamma}$, and maximum $\gamma_{max}$ to benchmark skill. Applied to high-resolution satellite SSI fields, the framework demonstrates robustness to minor displacements and isolates physically meaningful discrepancies, outperforming pixel-wise RMSE in representing usable forecasts. The approach is generic and transferable to any gridded variable, offering a practical complement to existing spatial verification tools with direct physical interpretation for operational decision-making.

Abstract

Classical field forecast evaluation relies mainly on local scores such as RMSE or MAE. These metrics severely over-penalize small spatial or temporal displacements of coherent structures, a limitation known as the double-penalty issue and common to many forecasting domains. The present paper introduces a tolerance-based framework built on the three-dimensional gamma index, initially designed for medical dose verification, as a unified acceptance criterion for gridded forecasts. The method embeds explicit margins in space (DTA), time (TTA), and intensity (IDT), and evaluates whether predictions agree with observations within predefined physical bounds rather than through pixel-wise differences only. A synthetic illustration is first used to show why conventional metrics can misrepresent usable forecasts. The approach is then applied to satellite-derived SSI fields to demonstrate operational behaviour on a real dataset. Results confirm that the gamma criterion preserves structural consistency under minor positional noise while isolating physically significant discrepancies. The formulation is generic and can be implemented for any gridded variable provided meaningful tolerances are defined, offering a pragmatic complement to existing spatial verification tools in general forecasting workflows.

A Tolerance-Based Framework for Spatio-Temporal Forecast Validation Using the gamma-Index

TL;DR

The paper addresses the problem of over-penalizing small spatial or temporal displacements in gridded forecast verification by introducing a three-dimensional -Index that jointly incorporates spatial (DTA), temporal (TTA), and intensity (IDT) tolerances. The method defines a normalized acceptance criterion, , as a minimum over spatial and temporal neighborhoods, and passes forecasts when , yielding metrics such as the passing rate , mean , and maximum to benchmark skill. Applied to high-resolution satellite SSI fields, the framework demonstrates robustness to minor displacements and isolates physically meaningful discrepancies, outperforming pixel-wise RMSE in representing usable forecasts. The approach is generic and transferable to any gridded variable, offering a practical complement to existing spatial verification tools with direct physical interpretation for operational decision-making.

Abstract

Classical field forecast evaluation relies mainly on local scores such as RMSE or MAE. These metrics severely over-penalize small spatial or temporal displacements of coherent structures, a limitation known as the double-penalty issue and common to many forecasting domains. The present paper introduces a tolerance-based framework built on the three-dimensional gamma index, initially designed for medical dose verification, as a unified acceptance criterion for gridded forecasts. The method embeds explicit margins in space (DTA), time (TTA), and intensity (IDT), and evaluates whether predictions agree with observations within predefined physical bounds rather than through pixel-wise differences only. A synthetic illustration is first used to show why conventional metrics can misrepresent usable forecasts. The approach is then applied to satellite-derived SSI fields to demonstrate operational behaviour on a real dataset. Results confirm that the gamma criterion preserves structural consistency under minor positional noise while isolating physically significant discrepancies. The formulation is generic and can be implemented for any gridded variable provided meaningful tolerances are defined, offering a pragmatic complement to existing spatial verification tools in general forecasting workflows.
Paper Structure (16 sections, 7 equations, 3 figures, 1 algorithm)

This paper contains 16 sections, 7 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Forecast Verification for Gridded Fields
  3. Gap, Problem Statement, and Proposed Contribution
  4. Materials and Methods
  5. Introduction of the $\gamma$-Index
  6. Metric Definition and Numerical Workflow
  7. Tolerance Selection and Implementation
  8. Results
  9. Synthetic Illustration
  10. HelioClim-3 Experiment
  11. Discussion
  12. Limitations
  13. Conclusions
  14. Beyond RMSE or MAE: A Spatiotemporal Perspective on Forecast Validation
  15. Variants of Classical Error Metrics in 2D–3D Forecast Workflows
  16. ...and 1 more sections

Figures (3)

  • Figure 1: 3D representation of the $\gamma$-Index tolerance region with DTA, TTA, and IDT. The black circles represent points inside the tolerance region ($\gamma \leq 1$), while black crosses represent points outside ($\gamma > 1$).
  • Figure 2: Comparison of mean pixel wise formulation of RMSE and the $\gamma$-Index for synthetic SSI validation. Black contours highlight areas where $\gamma>1$, indicating significant forecast errors.
  • Figure 3: Impact of spatio-temporal displacements and Gaussian noise on solar irradiance metrics. The figure presents (a) raw SSI measurements (July 4, 2002), (b) predicted SSI after spatial and temporal shifts, (c) RMSE map, (d) $\gamma$-Index map highlighting tolerance thresholds, (e) radar plot comparing $\gamma$-mean and $\gamma$-max across shifts, and (f) bar chart comparing RMSE and GPR validation across perturbations.