Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ACE Metric: Advection and Convection Evaluation for Accurate Weather Forecasting

Doyi Kim, Minseok Seo, Yeji Choi

TL;DR

The paper addresses the inadequacy of conventional metrics in evaluating data-driven weather forecasts, which often produce blur that inflates RMSE. It introduces the Advection and Convection Error (ACE) metric, decomposing forecast errors into advection (horizontal transport) and convection (vertical development) components via velocity-field estimation and a remapping step, with ACE = AE + CE/AE. Validations on WeatherBench2 and MovingMNIST demonstrate that ACE captures critical atmospheric dynamics, with methods explicitly modeling advection/convection achieving higher ACE and blurred outputs failing to improve ACE. The work provides a principled framework for assessing and guiding improvements in regional weather forecasting, potentially improving reliability for hazardous weather preparedness.

Abstract

Recently, data-driven weather forecasting methods have received significant attention for surpassing the RMSE performance of traditional NWP (Numerical Weather Prediction)-based methods. However, data-driven models are tuned to minimize the loss between forecasted data and ground truths, often using pixel-wise loss. This can lead to models that produce blurred outputs, which, despite being significantly different in detail from the actual weather conditions, still demonstrate low RMSE values. Although evaluation metrics from the computer vision field, such as PSNR, SSIM, and FVD, can be used, they are not entirely suitable for weather variables. This is because weather variables exhibit continuous physical changes over time and lack the distinct boundaries of objects typically seen in computer vision images. To resolve these issues, we propose the advection and convection Error (ACE) metric, specifically designed to assess how well models predict advection and convection, which are significant atmospheric transfer methods. We have validated the ACE evaluation metric on the WeatherBench2 and MovingMNIST datasets.

ACE Metric: Advection and Convection Evaluation for Accurate Weather Forecasting

TL;DR

The paper addresses the inadequacy of conventional metrics in evaluating data-driven weather forecasts, which often produce blur that inflates RMSE. It introduces the Advection and Convection Error (ACE) metric, decomposing forecast errors into advection (horizontal transport) and convection (vertical development) components via velocity-field estimation and a remapping step, with ACE = AE + CE/AE. Validations on WeatherBench2 and MovingMNIST demonstrate that ACE captures critical atmospheric dynamics, with methods explicitly modeling advection/convection achieving higher ACE and blurred outputs failing to improve ACE. The work provides a principled framework for assessing and guiding improvements in regional weather forecasting, potentially improving reliability for hazardous weather preparedness.

Abstract

Recently, data-driven weather forecasting methods have received significant attention for surpassing the RMSE performance of traditional NWP (Numerical Weather Prediction)-based methods. However, data-driven models are tuned to minimize the loss between forecasted data and ground truths, often using pixel-wise loss. This can lead to models that produce blurred outputs, which, despite being significantly different in detail from the actual weather conditions, still demonstrate low RMSE values. Although evaluation metrics from the computer vision field, such as PSNR, SSIM, and FVD, can be used, they are not entirely suitable for weather variables. This is because weather variables exhibit continuous physical changes over time and lack the distinct boundaries of objects typically seen in computer vision images. To resolve these issues, we propose the advection and convection Error (ACE) metric, specifically designed to assess how well models predict advection and convection, which are significant atmospheric transfer methods. We have validated the ACE evaluation metric on the WeatherBench2 and MovingMNIST datasets.
Paper Structure (18 sections, 8 equations, 5 figures, 2 tables)

This paper contains 18 sections, 8 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmark Dataset
  4. Data-driven Weather Forecasting
  5. Method
  6. Preliminaries
  7. Convection and Advection Error
  8. Advection Error
  9. Convection Error
  10. Experiments
  11. Datasets
  12. Evaluation metrics.
  13. Implementation details.
  14. MovingMNIST Results
  15. WeatherBench2 Results
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Comparison of RMSE, FVD, and ACE metrics in state-of-the-art deterministic, stochastic, and hybrid video prediction models.
  • Figure 2: Figure estimating Convection and Advection in infrared (IR) images from geostationary weather observation satellites. (c) involves extracting advection and warping the $T_{t2}$ image to generate the t1 image, excluding convection. (d) represents advection between t1 and t2, while (e) shows the difference between t2 and t1, and (f) illustrates the difference between the estimated t1 and the original t1. Note: There is a 10-hour difference between $T_{t1}$ and $T_{t2}$.
  • Figure 3: An illustration of advection visualized on the Moving MNIST dataset. The arrow direction represents t2 to t1. Bold indicates the highest score and underline indicates the second highest score.
  • Figure 4: The first row shows the prediction results and ground truth for a +5 days forecast. The second row presents a zoomed-in view of a specific region. The third row displays the RMSE map, the fourth row shows the AC map, and the fifth row depicts the CE map.r
  • Figure 5: The first row shows the prediction results and ground truth for a +10 days forecast. The second row displays the RMSE map, the third row shows the AC map, and the fourth row depicts the CE map.