Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sequential Scoring Rule Evaluation for Forecast Method Selection

David T. Frazier, Donald S. Poskitt

TL;DR

The paper addresses forecast method selection by introducing sequential scoring rule evaluation (SSRE), a framework that accumulates evidence from scoring-rule differences and terminates when boundaries are crossed. It develops a large-deviations-type analysis via a change of measure and reveals a deep link between SSRE and generalized e-values (GUe-values), enabling calibrated finite-sample error control. SSRE guarantees finite-time termination with moments of the stopping time existing, and it incorporates e-value based strategies to control tail probabilities and false discoveries across multiple comparisons. Empirical results demonstrate that SSRE with GUe-values delivers reliable, power-lean performance where traditional methods like the Diebold–Mariano test can fail, and the framework shows promise for extending to multi-model confidence sets in forecast evaluation.

Abstract

This paper shows that sequential statistical analysis techniques can be generalised to the problem of selecting between alternative forecasting methods using scoring rules. A return to basic principles is necessary in order to show that ideas and concepts from sequential statistical methods can be adapted and applied to sequential scoring rule evaluation (SSRE). One key technical contribution of this paper is the development of a large deviations type result for SSRE schemes using a change of measure that parallels a traditional exponential tilting form. Further, we also show that SSRE will terminate in finite time with probability one, and that the moments of the SSRE stopping time exist. A second key contribution is to show that the exponential tilting form underlying our large deviations result allows us to cast SSRE within the framework of generalised e-values. Relying on this formulation, we devise sequential testing approaches that are both powerful and maintain control on error probabilities underlying the analysis. Through several simulated examples, we demonstrate that our e-values based SSRE approach delivers reliable results that are more powerful than more commonly applied testing methods precisely in the situations where these commonly applied methods can be expected to fail.

Sequential Scoring Rule Evaluation for Forecast Method Selection

TL;DR

The paper addresses forecast method selection by introducing sequential scoring rule evaluation (SSRE), a framework that accumulates evidence from scoring-rule differences and terminates when boundaries are crossed. It develops a large-deviations-type analysis via a change of measure and reveals a deep link between SSRE and generalized e-values (GUe-values), enabling calibrated finite-sample error control. SSRE guarantees finite-time termination with moments of the stopping time existing, and it incorporates e-value based strategies to control tail probabilities and false discoveries across multiple comparisons. Empirical results demonstrate that SSRE with GUe-values delivers reliable, power-lean performance where traditional methods like the Diebold–Mariano test can fail, and the framework shows promise for extending to multi-model confidence sets in forecast evaluation.

Abstract

This paper shows that sequential statistical analysis techniques can be generalised to the problem of selecting between alternative forecasting methods using scoring rules. A return to basic principles is necessary in order to show that ideas and concepts from sequential statistical methods can be adapted and applied to sequential scoring rule evaluation (SSRE). One key technical contribution of this paper is the development of a large deviations type result for SSRE schemes using a change of measure that parallels a traditional exponential tilting form. Further, we also show that SSRE will terminate in finite time with probability one, and that the moments of the SSRE stopping time exist. A second key contribution is to show that the exponential tilting form underlying our large deviations result allows us to cast SSRE within the framework of generalised e-values. Relying on this formulation, we devise sequential testing approaches that are both powerful and maintain control on error probabilities underlying the analysis. Through several simulated examples, we demonstrate that our e-values based SSRE approach delivers reliable results that are more powerful than more commonly applied testing methods precisely in the situations where these commonly applied methods can be expected to fail.
Paper Structure (27 sections, 9 theorems, 64 equations, 2 figures, 1 table)

This paper contains 27 sections, 9 theorems, 64 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Motivation
  3. Background
  4. Organization of the Paper
  5. Preliminaries
  6. Scoring Rules
  7. Parameter Estimation
  8. Sequential Scoring Rule Evaluation and Termination
  9. SSRE Evaluation
  10. Termination of SSRE
  11. Boundary Values $k_l$ and $k_u$
  12. Implementation and Truncation
  13. Relationship to e-variables, and generalized e-variables
  14. SSRE and Generalized Universal e-values
  15. Implementation and False Discovery
  16. ...and 12 more sections

Key Result

Proposition 1

If under $\mathcal{H}_Q$ or $\mathcal{H}_P$ the distribution of $C_n(Q,P)=\prod_{m=1}^nR_m(Q,P)$ is nondegenerate and $\Pr(E^Q_n)\neq 0$ or $\Pr(E^P_n)\neq 0$, then for all $n\geq n_\epsilon+1$ there exists a $\varrho_\epsilon$ with $0<\varrho_\epsilon<1$ such that $\Pr(N>n)<\exp\{n_\epsilon\log(\va

Figures (2)

  • Figure 1: Regions of favorability for $\mathcal{H}_Q$ and $\mathcal{H}_P$ in terms of the SSRE boundary constants $k_l$ and $k_u$.
  • Figure 2: Regions of favorability for $\mathcal{H}_Q$ and $\mathcal{H}_P$ in terms of the modified SSRE boundary constants $k_l$ and $k_u$, with $k_l=1$.

Theorems & Definitions (13)

  • Proposition 1
  • Corollary 1
  • Lemma 1
  • proof
  • Proposition 2
  • Lemma 2
  • proof
  • Lemma 3
  • Theorem 1
  • proof
  • ...and 3 more