Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reducing the Impact of I/O Contention in Numerical Weather Prediction Workflows at Scale Using DAOS

Nicolau Manubens, Simon D. Smart, Emanuele Danovaro, Tiago Quintino, Adrian Jackson

TL;DR

The paper tackles the rising data and contention challenges in operational Numerical Weather Prediction (NWP) workflows by moving ECMWF's Field DataBase (FDB) from POSIX backends to DAOS-backed Catalogue and Store backends. It details the design and implementation of DAOS-based backends, leveraging MVCC and server-side contention resolution to achieve better performance under high I/O contention, and presents a thorough performance evaluation on NEXTGenIO hardware. Key findings show that DAOS backends deliver higher throughput and scalability in contended scenarios, with careful data-schema optimization (e.g., collocation handling and array-based storage) being crucial for peak performance. The work demonstrates DAOS as a viable, scalable storage substrate for ECMWF's data-intensive NWP pipelines and provides guidance for deploying similar semantically driven I/O systems in HPC environments.

Abstract

Operational Numerical Weather Prediction (NWP) workflows are highly data-intensive. Data volumes have increased by many orders of magnitude over the last 40 years, and are expected to continue to do so, especially given the upcoming adoption of Machine Learning in forecast processes. Parallel POSIX-compliant file systems have been the dominant paradigm in data storage and exchange in HPC workflows for many years. This paper presents ECMWF's move beyond the POSIX paradigm, implementing a backend for their storage library to support DAOS -- a novel high-performance object store designed for massively distributed Non-Volatile Memory. This system is demonstrated to be able to outperform the highly mature and optimised POSIX backend when used under high load and contention, as per typical forecast workflow I/O patterns. This work constitutes a significant step forward, beyond the performance constraints imposed by POSIX semantics.

Reducing the Impact of I/O Contention in Numerical Weather Prediction Workflows at Scale Using DAOS

TL;DR

The paper tackles the rising data and contention challenges in operational Numerical Weather Prediction (NWP) workflows by moving ECMWF's Field DataBase (FDB) from POSIX backends to DAOS-backed Catalogue and Store backends. It details the design and implementation of DAOS-based backends, leveraging MVCC and server-side contention resolution to achieve better performance under high I/O contention, and presents a thorough performance evaluation on NEXTGenIO hardware. Key findings show that DAOS backends deliver higher throughput and scalability in contended scenarios, with careful data-schema optimization (e.g., collocation handling and array-based storage) being crucial for peak performance. The work demonstrates DAOS as a viable, scalable storage substrate for ECMWF's data-intensive NWP pipelines and provides guidance for deploying similar semantically driven I/O systems in HPC environments.

Abstract

Operational Numerical Weather Prediction (NWP) workflows are highly data-intensive. Data volumes have increased by many orders of magnitude over the last 40 years, and are expected to continue to do so, especially given the upcoming adoption of Machine Learning in forecast processes. Parallel POSIX-compliant file systems have been the dominant paradigm in data storage and exchange in HPC workflows for many years. This paper presents ECMWF's move beyond the POSIX paradigm, implementing a backend for their storage library to support DAOS -- a novel high-performance object store designed for massively distributed Non-Volatile Memory. This system is demonstrated to be able to outperform the highly mature and optimised POSIX backend when used under high load and contention, as per typical forecast workflow I/O patterns. This work constitutes a significant step forward, beyond the performance constraints imposed by POSIX semantics.
Paper Structure (21 sections, 6 figures)

This paper contains 21 sections, 6 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Numerical Weather Prediction Data Flows
  4. The FDB
  5. The Distributed Asynchronous Object Store (DAOS)
  6. Design of the DAOS FDB Backends
  7. The Store Backend
  8. Interface
  9. Implementation
  10. The Catalogue Backend
  11. Interface
  12. Implementation
  13. Performance assessment
  14. Test system
  15. Benchmark
  16. ...and 6 more sections

Figures (6)

  • Figure 1: A simplified overview of ECMWF's operational work flow
  • Figure 2: Diagram of DAOS entities resulting from an FDB archive call to store and index the first weather field in a simulation.
  • Figure 3: Bandwidth results for fdb-hammer/DAOS (8 server nodes; top row) and fdb-hammer/Lustre (8+1 server nodes; bottom row) runs. No w+r contention.
  • Figure 4: Bandwidths observed in short fdb-hammer and Field I/O runs without and with write/read contention.
  • Figure 5: Profiling results for fdb-hammer/DAOS runs (12 server and 20 client nodes; 32 processes per client node).
  • ...and 1 more figures