Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TorchDA: A Python package for performing data assimilation with deep learning forward and transformation functions

Sibo Cheng, Jinyang Min, Che Liu, Rossella Arcucci

TL;DR

A novel Python package seamlessly combining data assimilation with deep neural networks to serve as models for state transition and observation functions is presented, conferring a versatile tool to tackle complex high dimensional dynamical systems across scientific domains.

Abstract

Data assimilation techniques are often confronted with challenges handling complex high dimensional physical systems, because high precision simulation in complex high dimensional physical systems is computationally expensive and the exact observation functions that can be applied in these systems are difficult to obtain. It prompts growing interest in integrating deep learning models within data assimilation workflows, but current software packages for data assimilation cannot handle deep learning models inside. This study presents a novel Python package seamlessly combining data assimilation with deep neural networks to serve as models for state transition and observation functions. The package, named TorchDA, implements Kalman Filter, Ensemble Kalman Filter (EnKF), 3D Variational (3DVar), and 4D Variational (4DVar) algorithms, allowing flexible algorithm selection based on application requirements. Comprehensive experiments conducted on the Lorenz 63 and a two-dimensional shallow water system demonstrate significantly enhanced performance over standalone model predictions without assimilation. The shallow water analysis validates data assimilation capabilities mapping between different physical quantity spaces in either full space or reduced order space. Overall, this innovative software package enables flexible integration of deep learning representations within data assimilation, conferring a versatile tool to tackle complex high dimensional dynamical systems across scientific domains.

TorchDA: A Python package for performing data assimilation with deep learning forward and transformation functions

TL;DR

A novel Python package seamlessly combining data assimilation with deep neural networks to serve as models for state transition and observation functions is presented, conferring a versatile tool to tackle complex high dimensional dynamical systems across scientific domains.

Abstract

Data assimilation techniques are often confronted with challenges handling complex high dimensional physical systems, because high precision simulation in complex high dimensional physical systems is computationally expensive and the exact observation functions that can be applied in these systems are difficult to obtain. It prompts growing interest in integrating deep learning models within data assimilation workflows, but current software packages for data assimilation cannot handle deep learning models inside. This study presents a novel Python package seamlessly combining data assimilation with deep neural networks to serve as models for state transition and observation functions. The package, named TorchDA, implements Kalman Filter, Ensemble Kalman Filter (EnKF), 3D Variational (3DVar), and 4D Variational (4DVar) algorithms, allowing flexible algorithm selection based on application requirements. Comprehensive experiments conducted on the Lorenz 63 and a two-dimensional shallow water system demonstrate significantly enhanced performance over standalone model predictions without assimilation. The shallow water analysis validates data assimilation capabilities mapping between different physical quantity spaces in either full space or reduced order space. Overall, this innovative software package enables flexible integration of deep learning representations within data assimilation, conferring a versatile tool to tackle complex high dimensional dynamical systems across scientific domains.
Paper Structure (23 sections, 13 equations, 12 figures, 4 tables, 4 algorithms)

This paper contains 23 sections, 13 equations, 12 figures, 4 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Data Assimilation Concept
  4. Kalman Filter
  5. Ensemble Kalman Filter
  6. 3D Variational data assimilation
  7. 4D Variational data assimilation
  8. Integrating Data Assimilation and Deep Learning
  9. Package Structure
  10. Computational Examples
  11. Lorenz 63 Model
  12. Experiment without Data Assimilation
  13. Experiment of EnKF
  14. Experiment of 4DVar
  15. Shallow Water Model
  16. ...and 8 more sections

Figures (12)

  • Figure 1: Statistical Data of Deep Learning and Data Assimilation Combined Applications, and Interests Trend of Popular Machine Learning Frameworks on Google
  • Figure 2: Data Assimilation Algorithms
  • Figure 3: Schematic Diagram of the Latent Data Assimilation with Deep Neural Network Surrogate Models
  • Figure 4: Package Structure and Dependencies
  • Figure 5: Reference Trajectory with Noisy Observations
  • ...and 7 more figures