Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kilometer-Scale E3SM Land Model Simulation over North America

Dali Wang, Chen Wang, Qinglei Cao, Peter Schwartz, Fengming Yuan, Jayesh Krishna, Danqing Wu, Danial Ricciuto, Peter Thornton, Shih-Chieh Kao, Michele Thornton, Kathryn Mohror

TL;DR

This paper demonstrates kilometer-scale ELM in the E3SM framework, addressing the challenge of high-resolution land modeling on exascale systems over North America. It integrates a data-driven configuration using CIME and the KiloCraft toolkit to generate 1 km forcing and surface datasets, and validates performance through AKSP baseline and domain-duplication experiments on Summit, achieving up to 100,800 cores. Key contributions include the first full-capacity km-scale continental simulations, a scalable data-management workflow with PIO/SCORPIO/I/O aggregation, and comprehensive strong and weak scaling assessments showing substantial efficiency and IO considerations. The work enables future fully-coupled, high-resolution E3SM simulations and area-of-interest studies, with plans to extend to GPU acceleration and richer surface-property datasets for improved realism and throughput.

Abstract

The development of a kilometer-scale E3SM Land Model (km-scale ELM) is an integral part of the E3SM project, which seeks to advance energy-related Earth system science research with state-of-the-art modeling and simulation capabilities on exascale computing systems. Through the utilization of high-fidelity data products, such as atmospheric forcing and soil properties, the km-scale ELM plays a critical role in accurately modeling geographical characteristics and extreme weather occurrences. The model is vital for enhancing our comprehension and prediction of climate patterns, as well as their effects on ecosystems and human activities. This study showcases the first set of full-capability, km-scale ELM simulations over various computational domains, including simulations encompassing 21.6 million land gridcells, reflecting approximately 21.5 million square kilometers of North America at a 1 km x 1 km resolution. We present the largest km-scale ELM simulation using up to 100,800 CPU cores across 2,400 nodes. This continental-scale simulation is 300 times larger than any previous studies, and the computational resources used are about 400 times larger than those used in prior efforts. Both strong and weak scaling tests have been conducted, revealing exceptional performance efficiency and resource utilization. The km-scale ELM uses the common E3SM modeling infrastructure and a general data toolkit known as KiloCraft. Consequently, it can be readily adapted for both fully-coupled E3SM simulations and data-driven simulations over specific areas, ranging from a single gridcell to the entire North America.

Kilometer-Scale E3SM Land Model Simulation over North America

TL;DR

This paper demonstrates kilometer-scale ELM in the E3SM framework, addressing the challenge of high-resolution land modeling on exascale systems over North America. It integrates a data-driven configuration using CIME and the KiloCraft toolkit to generate 1 km forcing and surface datasets, and validates performance through AKSP baseline and domain-duplication experiments on Summit, achieving up to 100,800 cores. Key contributions include the first full-capacity km-scale continental simulations, a scalable data-management workflow with PIO/SCORPIO/I/O aggregation, and comprehensive strong and weak scaling assessments showing substantial efficiency and IO considerations. The work enables future fully-coupled, high-resolution E3SM simulations and area-of-interest studies, with plans to extend to GPU acceleration and richer surface-property datasets for improved realism and throughput.

Abstract

The development of a kilometer-scale E3SM Land Model (km-scale ELM) is an integral part of the E3SM project, which seeks to advance energy-related Earth system science research with state-of-the-art modeling and simulation capabilities on exascale computing systems. Through the utilization of high-fidelity data products, such as atmospheric forcing and soil properties, the km-scale ELM plays a critical role in accurately modeling geographical characteristics and extreme weather occurrences. The model is vital for enhancing our comprehension and prediction of climate patterns, as well as their effects on ecosystems and human activities. This study showcases the first set of full-capability, km-scale ELM simulations over various computational domains, including simulations encompassing 21.6 million land gridcells, reflecting approximately 21.5 million square kilometers of North America at a 1 km x 1 km resolution. We present the largest km-scale ELM simulation using up to 100,800 CPU cores across 2,400 nodes. This continental-scale simulation is 300 times larger than any previous studies, and the computational resources used are about 400 times larger than those used in prior efforts. Both strong and weak scaling tests have been conducted, revealing exceptional performance efficiency and resource utilization. The km-scale ELM uses the common E3SM modeling infrastructure and a general data toolkit known as KiloCraft. Consequently, it can be readily adapted for both fully-coupled E3SM simulations and data-driven simulations over specific areas, ranging from a single gridcell to the entire North America.
Paper Structure (25 sections, 6 figures, 6 tables)

This paper contains 25 sections, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Km-scale ELM simulation
  4. Method
  5. Data-driven, km-scale ELM configuration and simulation
  6. Component configuration and dataflow
  7. Computational domain, domain decomposition, and parallel execution
  8. I/O and data management
  9. Km-scale ELM input data generation
  10. ELM domain over North America
  11. ELM forcing data
  12. ELM surface properties
  13. ELM experiments to evaluate massive simulation readiness
  14. AKSP baseline simulation
  15. Domain duplication for simulation readiness evaluation
  16. ...and 10 more sections

Figures (6)

  • Figure 1: Km-scale ELM data flow and simulation configuration, which requires three groups of datastreams: atmospheric forcing, land surface properties, and others (e.g., nitrogen and phosphorus (N/P) deposition).
  • Figure 2: Samples of 1 km by 1 km input data across North America. Top left: humidity (01/01/2014); top right: temperature (01/01/2014); bottom left maximum fractional saturated area (percentage); bottom right: clay (percentage) in soil.
  • Figure 3: The domain and input dataset of the AKSP reference simulation. Top left: humidity (01/01/2014); top right: temperature (01/01/2014); bottom left: maximum fractional saturated area (percentage); bottom right: clay (percentage) in the soil.
  • Figure 4: Visualization of output variables from the experiment. Top left: ground coverage by snow; top right: water in soil; bottom left: total leaf area index; bottom right: soil temperature.
  • Figure 5: Speedup of the land model in the strong scaling experiments.
  • ...and 1 more figures