Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Computing the Full Earth System at 1 km Resolution

Daniel Klocke, Claudia Frauen, Jan Frederik Engels, Dmitry Alexeev, René Redler, Reiner Schnur, Helmuth Haak, Luis Kornblueh, Nils Brüggemann, Fatemeh Chegini, Manoel Römmer, Lars Hoffmann, Sabine Griessbach, Mathis Bode, Jonathan Coles, Miguel Gila, William Sawyer, Alexandru Calotoiu, Yakup Budanaz, Pratyai Mazumder, Marcin Copik, Benjamin Weber, Andreas Herten, Hendryk Bockelmann, Torsten Hoefler, Cathy Hohenegger, Bjorn Stevens

TL;DR

This work demonstrates for the first time a global, fully coupled Earth system simulation at $1.25$-km resolution with substantial temporal compression, achieved by leveraging heterogeneous GH200-based GPUs and CPUs. The authors map atmosphere/land to GPUs while placing ocean/biogeochemistry on CPUs, coordinated via a coupling interface, and they separate concerns by translating the Fortran dynamical core into a dataflow representation with DaCe, enabling architecture-specific optimizations without touching domain code. Key contributions include a final component mapping strategy, a DaCe-based Fortran-to-GPU workflow that yields large reductions in index lookups and memory bandwidth improvements, and strong/weak scaling results that reach up to $\\tau=145.7$ simulated days per day on high-end systems. The result is a scalable path to year-to-decade, full-Earth-system studies at km-scale, with significant implications for understanding climate feedbacks and informing adaptation under future warming scenarios, while illustrating a productive collaboration model between domain scientists and computer scientists for exascale-ready climate computing.

Abstract

We present the first-ever global simulation of the full Earth system at 1.25 km grid spacing, achieving highest time compression with an unseen number of degrees of freedom. Our model captures the flow of energy, water, and carbon through key components of the Earth system: atmosphere, ocean, and land. To achieve this landmark simulation, we harness the power of 8192 GPUs on Alps and 20480 GPUs on JUPITER, two of the world's largest GH200 superchip installations. We use both the Grace CPUs and Hopper GPUs by carefully balancing Earth's components in a heterogeneous setup and optimizing acceleration techniques available in ICON's codebase. We show how separation of concerns can reduce the code complexity by half while increasing performance and portability. Our achieved time compression of 145.7 simulated days per day enables long studies including full interactions in the Earth system and even outperforms earlier atmosphere-only simulations at a similar resolution.

Computing the Full Earth System at 1 km Resolution

TL;DR

This work demonstrates for the first time a global, fully coupled Earth system simulation at -km resolution with substantial temporal compression, achieved by leveraging heterogeneous GH200-based GPUs and CPUs. The authors map atmosphere/land to GPUs while placing ocean/biogeochemistry on CPUs, coordinated via a coupling interface, and they separate concerns by translating the Fortran dynamical core into a dataflow representation with DaCe, enabling architecture-specific optimizations without touching domain code. Key contributions include a final component mapping strategy, a DaCe-based Fortran-to-GPU workflow that yields large reductions in index lookups and memory bandwidth improvements, and strong/weak scaling results that reach up to simulated days per day on high-end systems. The result is a scalable path to year-to-decade, full-Earth-system studies at km-scale, with significant implications for understanding climate feedbacks and informing adaptation under future warming scenarios, while illustrating a productive collaboration model between domain scientists and computer scientists for exascale-ready climate computing.

Abstract

We present the first-ever global simulation of the full Earth system at 1.25 km grid spacing, achieving highest time compression with an unseen number of degrees of freedom. Our model captures the flow of energy, water, and carbon through key components of the Earth system: atmosphere, ocean, and land. To achieve this landmark simulation, we harness the power of 8192 GPUs on Alps and 20480 GPUs on JUPITER, two of the world's largest GH200 superchip installations. We use both the Grace CPUs and Hopper GPUs by carefully balancing Earth's components in a heterogeneous setup and optimizing acceleration techniques available in ICON's codebase. We show how separation of concerns can reduce the code complexity by half while increasing performance and portability. Our achieved time compression of 145.7 simulated days per day enables long studies including full interactions in the Earth system and even outperforms earlier atmosphere-only simulations at a similar resolution.

Paper Structure

This paper contains 13 sections, 5 figures, 3 tables.

Table of Contents

  1. Overview of the Problem
  2. Current State of the Art
  3. Innovations Realized
  4. Exploiting the Earth System's Complexity to Flexibly Utilize System Parallelism
  5. Final Mapping
  6. Separation of Concerns: From Fortran to GPU
  7. How Performance Was Measured
  8. Earth System Model Configuration
  9. System Details
  10. Performance Metrics
  11. I/O Performance Metrics
  12. Performance Results
  13. Implications

Figures (5)

  • Figure 1: Schematic showing the main components of the Earth system in ICON and the exchange of energy, water, and carbon among them. Table \ref{['tab_resolution']} presents the degrees of freedom assigned to each component.
  • Figure 2: Strong scaling on CPUs and GPUs of the coupled 10 km ICON (without biogeochemistry submodel) on Levante (left). Nearly four-fold improved energy efficiency on Levante GPUs vs CPUs (right).
  • Figure 3: Mapping of the Earth system components (from Figure \ref{['fig:ICON-Modell']}) to the GH200 superchip with indicated innovations. On the platforms tested, four superchips are on each node. For our benchmarking experiments, we utilized 10 MPI processes with each 6 OpenMP threads for the ocean on the CPUs and 1 MPI process for the GPU of each GH200 superchip.
  • Figure 4: Strong scaling of the full ICON Earth system model at 1.25 grid-spacing (left). The gray line shows scaling with a configuration with a factor 64 less grid cells (10 resolution) as the 1.25 configuration but using the same timestep, serving as a reference line assuming perfect weak-scaling. Strong scaling of the ICON Earth system model with 10 grid-spacing on Alps and JEDI (right)
  • Figure 5: Snapshot of phytoplankton, near-surface wind and air-sea CO$_{2}$ flux at 2020-01-01 03:00. On the left, phytoplankton concentration on a logarithmic scale between e-9 and e-6 are shown. In the center, surface winds are displayed with color limits in the range from 020. On the right, the air-sea/land carbon flux is illustrated with the color scale spanning $\pm4\cdot10^{-7}$kg m$^{-2}$ s$^{-1}$ and green values implying carbon uptake and blue values indicating carbon release for ocean and land (values over the ocean were multiplied by 30 to enhance visibility). Insets highlight specific regions: water mass upwelling off the coast of Chile (left), near-surface wind patterns over the mountains of the Balkans (center), and CO$_2$ flux off the coast of Tasmania (right).