Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating How Neighbourhood Scores Reflect Forecast Error

Bobby Antonio

Abstract

Meaningful scores for forecast verification are essential for developing reliable forecasts, and there has been much effort to develop scores that align well with human perceptions of forecast quality. Whilst many of these scores have intuitive interpretations, relatively little is known about how these scores rank different forecasts, and how scores reflect forecast error. We theoretically explore the behaviour of two scores that fall within the `neighbourhood' paradigm of spatial verification; the Fractions Skill Score (FSS) and Brier Divergence Skill Score (BDnSS). We investigate how each score ranks forecasts with two types of error; errors in the mean frequency (corresponding to intensity or shape errors) and errors in the standard deviation (corresponding to errors in spatial structure, such as blurring or excess noise). We find that under many situations the FSS assigns higher scores to forecasts that over-predict mean frequency, thus theoretically confirming the need to use the FSS with percentile thresholds. Both scores assign higher scores to smoother forecasts in many situations, a reflection of the `double penalty' problem; however, we observe that size of this effect is larger for the BDnSS than the FSS, showing that the FSS under some situations is less susceptible to the double penalty problem than the BDnSS.

Investigating How Neighbourhood Scores Reflect Forecast Error

Abstract

Meaningful scores for forecast verification are essential for developing reliable forecasts, and there has been much effort to develop scores that align well with human perceptions of forecast quality. Whilst many of these scores have intuitive interpretations, relatively little is known about how these scores rank different forecasts, and how scores reflect forecast error. We theoretically explore the behaviour of two scores that fall within the `neighbourhood' paradigm of spatial verification; the Fractions Skill Score (FSS) and Brier Divergence Skill Score (BDnSS). We investigate how each score ranks forecasts with two types of error; errors in the mean frequency (corresponding to intensity or shape errors) and errors in the standard deviation (corresponding to errors in spatial structure, such as blurring or excess noise). We find that under many situations the FSS assigns higher scores to forecasts that over-predict mean frequency, thus theoretically confirming the need to use the FSS with percentile thresholds. Both scores assign higher scores to smoother forecasts in many situations, a reflection of the `double penalty' problem; however, we observe that size of this effect is larger for the BDnSS than the FSS, showing that the FSS under some situations is less susceptible to the double penalty problem than the BDnSS.
Paper Structure (10 sections, 30 equations, 3 figures)

This paper contains 10 sections, 30 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Neighbourhoods definitions
  3. The Double Penalty problem
  4. Definition of scores
  5. Fractions Skill Score
  6. Brier Divergence Skill Score
  7. Effects of errors on the scores
  8. Effect of errors in the event frequency
  9. Effects of errors in the neighbourhood standard deviation
  10. Discussion and Conclusions

Figures (3)

  • Figure 1: Analysis of the frequency error that maximises the FSS, for a range of values of the neighbourhood correlation $r_n$ and coefficient of variation $C$ (a) value of multiplicative error that maximises the FSS, where an ideal verification score has $R_{\mu, max}=1$ (b) difference between the maximum value of FSS, and the FSS with no error in the frequency (i.e. $R_{\mu}=1$).
  • Figure 2: Comparison of values of $R_\sigma$ that maximise the FSS and BDnSS scores, for different values of the coefficient of variation $C$ and neighbourhood correlation $r_n$ (a) the value of $R_\sigma$ that maximises the FSS (b) the difference in $R_{\sigma, max}^{(\text{FSS})}$ and $R_{\sigma, max}^{(\text{BDnSS})}$, where more positive values indicate where the FSS is less susceptible to the double penalty problem than the BDnSS.
  • Figure 3: Analysis of difference between maximised scores, and scores with no error in the neighbourhood standard deviation (i.e. $R_{\sigma}=1$) (a) difference for the FSS for different values of $C$ and $r_n$ (b) difference for the BDnSS, for different values of $r_n$.