evalprob4cast: An R-package for evaluation of ensembles as probabilistic forecasts or event forecasts

TL;DR

The paper introduces evalprob4cast, an R-package that unifies evaluation of ensemble forecasts as probabilistic densities and as event forecasts, aligning with IEA Wind guidelines. It implements state-of-the-art metrics—CRPS, LogS, VarS, ranked histograms, Brier scores, reliability diagrams, ROC curves, and contingency tables—and provides an event-detection workflow via event_detection_table, enabling flexible, scalable assessment for univariate and multivariate forecasts. Through practical examples with wind-forecast datasets, the authors demonstrate rapid, end-to-end evaluation workflows, including density evaluation, lead-time analysis, and event-based performance across models. The package aims to fill a gap in open-source tools for ensemble evaluation, offering both high-level usability and low-level access for customized analyses, with ongoing development and community contributions encouraged.

Abstract

For any forecasting application, evaluation of forecasts is an important task. For example, in the field of renewable energy sources there is high variability and uncertainty of power production, which makes forecasting and the evaluation hereof crucial both for power trading and power grid balancing. In particular, probabilistic forecasts represented by ensembles are popular due to their ability to cover the full range of scenarios that can occur, thus enabling forecast users to make more informed decisions than what would be possible with simple deterministic forecasts. The selection of open source software that supports evaluation of ensemble forecasts, and especially event detection, is currently limited. As a solution, evalprob4cast is a new R-package for probabilistic forecast evaluation that aims to provide its users with all the tools needed for the assessment of ensemble forecasts, in the form of metrics and visualization methods. Both univariate and multivariate probabilistic forecasts as well as event detection are covered. Furthermore, it offers a user-friendly design where all of the evaluation methods can be applied in a fast and easy way, as long as the input data is organized in accordance with the format defined by the package. While its development is motivated by forecasting of renewables, the package can be used for any application with ensemble forecasts.

Figures (13)

• Figure 1: Examples of event search.
• Figure 2: Example of a rolling window of event searches.
• Figure 3: A reliability diagram.
• Figure 4: A ROC curve.
• Figure 5: The path from data structure to forecast evaluation. Data is shown as white nodes and evalprob4cast functions are shown as light blue nodes.