Not a fragment, but the whole: Map-based evaluation of data-driven Fire Danger Index models

Abstract

A growing body of literature has focused on predicting wildfire occurrence using machine learning methods, capitalizing on high-resolution data and fire predictors that canonical process-based frameworks largely ignore. Standard evaluation metrics for an ML classifier, while important, provide a potentially limited measure of the model's operational performance for the Fire Danger Index (FDI) forecast. Furthermore, model evaluation is frequently conducted without adequately accounting for false positive rates, despite their critical relevance in operational contexts. In this paper, we revisit the daily FDI model evaluation paradigm and propose a novel method for evaluating a forest fire forecasting model that is aligned with real-world decision-making. Furthermore, we systematically assess performance in accurately predicting fire activity and the false positives (false alarms). We further demonstrate that an ensemble of ML models improves both fire identification and reduces false positives.

Figures (8)

• Figure 1: Geographic map of the target region. The region is centered on Greece, including the Balkan peninsula and western Turkey.
• Figure 2: (a) Figure shows the format of a single sample for the CNN network, which is a 3D tensor. (b) A single sample of the ConvLSTM network also includes a time dimension of length 10, and it is a 4D tensor.
• Figure 3: Plot showing the distribution of the fires in the entire dataset by their CLC class. On the x-axis are the CLC labels, and on the y-axis is the frequency of fire activity with that CLC class. The orange colored bars represent the distribution of the fire samples used for model training.
• Figure 4: The distribution of the FDI values on the map on no-fire days randomly selected for the analysis. A positively skewed FDI distribution indicates fewer false positives on that day compared to a negatively skewed distribution. ConvLSTM generally has a more positively skewed distribution than the CNN architectures.
• Figure 5: Full-map inference for the no-fire days randomly selects for the analysis (the same days as in Figure \ref{['fig:fdi_dist_nofire']}). Despite some differences between the maps, similarities can also be appreciated, indicating that each model is a noisy estimator of the underlying true FDI.