Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Regularization of ML models for Earth systems by using longer model timesteps

Raghul Parthipan, Mohit Anand, Hannah M Christensen, Frederic Vitart, Damon J Wischik, Jakob Zscheischler

Abstract

Regularization is a technique to improve generalization of machine learning (ML) models. A common form of regularization in the ML literature is to train on data where similar inputs map to different outputs. This improves generalization by preventing ML models from becoming overconfident in their predictions. This paper shows how using longer timesteps when modelling chaotic Earth systems naturally leads to more of this regularization. We show this in two domains. We explain how using longer model timesteps can improve results and demonstrate that increased regularization is one of the causes. We explain why longer model timesteps lead to improved regularization in these systems and present a procedure to pick the model timestep. We also carry out a benchmarking exercise on ORAS5 ocean reanalysis data to show that a longer model timestep (28 days) than is typically used gives realistic simulations. We suggest that there will be many opportunities to use this type of regularization in Earth system problems because the Earth system is chaotic and the regularization is so easy to implement.

Regularization of ML models for Earth systems by using longer model timesteps

Abstract

Regularization is a technique to improve generalization of machine learning (ML) models. A common form of regularization in the ML literature is to train on data where similar inputs map to different outputs. This improves generalization by preventing ML models from becoming overconfident in their predictions. This paper shows how using longer timesteps when modelling chaotic Earth systems naturally leads to more of this regularization. We show this in two domains. We explain how using longer model timesteps can improve results and demonstrate that increased regularization is one of the causes. We explain why longer model timesteps lead to improved regularization in these systems and present a procedure to pick the model timestep. We also carry out a benchmarking exercise on ORAS5 ocean reanalysis data to show that a longer model timestep (28 days) than is typically used gives realistic simulations. We suggest that there will be many opportunities to use this type of regularization in Earth system problems because the Earth system is chaotic and the regularization is so easy to implement.

Paper Structure

This paper contains 44 sections, 10 equations, 12 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Similar input -- different output (SIDO) regularization
  4. Classification
  5. Regression
  6. Generative modelling
  7. Multitask learning
  8. Regularization to improve roll-outs in ML weather forecasts
  9. Can a longer model timestep be helpful?
  10. Lorenz 96
  11. Dataset
  12. Model
  13. Experiments
  14. ORAS5 ocean reanalysis
  15. Dataset
  16. ...and 29 more sections

Figures (12)

  • Figure 1: Effect of model timestep on Lorenz 96 skill scores. This compares the skill (y-axis) in terms of CRPS ((a)) and RMSE ((b)) as a function of timestep (x-axis) of nine separate models. Lines join model skill scores for a particular forecast lead time. The darkest line is for the $10 \times 0.005$ model time unit lead time, and the lightest line is for the $4000 \times 0.005$ model time unit lead time. In all cases, lower is better.
  • Figure 2: Effect of model timestep on Lorenz 96 spread/skill scores. This compares the spread/skill (y-axis) as a function of timestep (x-axis) of nine separate models. Lines join model skill scores for a particular forecast lead time. The darkest line is for the $10 \times 0.005$ model time unit lead time, and the lightest line is for the $4000 \times 0.005$ model time unit lead time. In all cases, a score closer to one is better.
  • Figure 3: Effect of model timestep on ORAS5 skill scores for 2019-2022. This compares the skill (y-axis) in terms of CRPS ((a), (c), (e)) and RMSE ((b), (d), (f)) as a function of timestep (x-axis) of seven separate models (timesteps 1, 2, 7, 14, 28, 56 and 84 days). Skill is evaluated for Nino3.4 SST, whole-ocean SST, and whole-ocean SSH. Lines join model skill scores for a particular forecast lead time. The darkest line is for the 28-day lead time, and the lightest line is for the 196-day lead time. Skill scores for the 84-timestep model are only available at lead times of 84 and 168. In all cases, lower is better. The x-axis is logarithmic.
  • Figure 4: Effect of model timestep on ORAS5 spread/skill scores for 2019-2022. This compares the spread/skill (y-axis) as a function of timestep (x-axis) of seven separate models (timesteps 1, 2, 7, 14, 28, 56 and 84 days). Spread/skill is evaluated for Nino3.4 SST, whole-ocean SST, and whole-ocean SSH. Lines join model skill scores for a particular forecast lead time. The darkest line is for the 28-day lead time, and the lightest line is for the 196-day lead time. Skill scores for the 84-timestep model are only available at lead times of 84 and 168. In all cases, a score closer to one is better. The x-axis is logarithmic.
  • Figure 5: Effect of model timestep on long-run stability. For each model, a 51-member ensemble was run out for 4700 roll-out iterations (y-axis). At each iteration, the median and interquartile range of the minimum sea surface temperature (SST) across ensemble members is plotted in purple. Similarly, the median and interquartile range of the maximum SST across ensemble members is plotted in orange. The SST x-axis is logarithmic.
  • ...and 7 more figures