Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ensemble score filter with image inpainting for data assimilation in tracking surface quasi-geostrophic dynamics with partial observations

Siming Liang, Hoang Tran, Feng Bao, Hristo G. Chipilski, Peter Jan van Leeuwen, Guannan Zhang

TL;DR

The paper tackles data assimilation for high-dimensional, nonlinear, and partially observed geophysical systems using an ensemble score filter (EnSF) that integrates training-free diffusion-based score modeling with image inpainting. At each step, EnSF maps the prior to a Gaussian via forward–backward SDEs, updates the observed state using a likelihood-driven score, and reconstructs unobserved components with PDE-based or dictionary-learning inpainting. Empirical results on the surface quasi-geostrophic (SQG) model show EnSF with inpainting outperforms the traditional LETKF in nonlinear and sparse-observation scenarios, with dictionary-based inpainting excelling under extreme observation sparsity and PDE-based methods performing well when data are more plentiful. The approach is training-free, scalable, and adaptable, offering a practical framework for improved data assimilation in operational geoscience and weather forecasting, with potential extensions to more complex models and adaptive inpainting strategies.

Abstract

Data assimilation plays a pivotal role in understanding and predicting turbulent systems within geoscience and weather forecasting, where data assimilation is used to address three fundamental challenges, i.e., high-dimensionality, nonlinearity, and partial observations. Recent advances in machine learning (ML)-based data assimilation methods have demonstrated encouraging results. In this work, we develop an ensemble score filter (EnSF) that integrates image inpainting to solve the data assimilation problems with partial observations. The EnSF method exploits an exclusively designed training-free diffusion models to solve high-dimensional nonlinear data assimilation problems. Its performance has been successfully demonstrated in the context of having full observations, i.e., all the state variables are directly or indirectly observed. However, because the EnSF does not use a covariance matrix to capture the dependence between the observed and unobserved state variables, it is nontrivial to extend the original EnSF method to the partial observation scenario. In this work, we incorporate various image inpainting techniques into the EnSF to predict the unobserved states during data assimilation. At each filtering step, we first use the diffusion model to estimate the observed states by integrating the likelihood information into the score function. Then, we use image inpainting methods to predict the unobserved state variables. We demonstrate the performance of the EnSF with inpainting by tracking the Surface Quasi-Geostrophic (SQG) model dynamics under a variety of scenarios. The successful proof of concept paves the way to more in-depth investigations on exploiting modern image inpainting techniques to advance data assimilation methodology for practical geoscience and weather forecasting problems.

Ensemble score filter with image inpainting for data assimilation in tracking surface quasi-geostrophic dynamics with partial observations

TL;DR

The paper tackles data assimilation for high-dimensional, nonlinear, and partially observed geophysical systems using an ensemble score filter (EnSF) that integrates training-free diffusion-based score modeling with image inpainting. At each step, EnSF maps the prior to a Gaussian via forward–backward SDEs, updates the observed state using a likelihood-driven score, and reconstructs unobserved components with PDE-based or dictionary-learning inpainting. Empirical results on the surface quasi-geostrophic (SQG) model show EnSF with inpainting outperforms the traditional LETKF in nonlinear and sparse-observation scenarios, with dictionary-based inpainting excelling under extreme observation sparsity and PDE-based methods performing well when data are more plentiful. The approach is training-free, scalable, and adaptable, offering a practical framework for improved data assimilation in operational geoscience and weather forecasting, with potential extensions to more complex models and adaptive inpainting strategies.

Abstract

Data assimilation plays a pivotal role in understanding and predicting turbulent systems within geoscience and weather forecasting, where data assimilation is used to address three fundamental challenges, i.e., high-dimensionality, nonlinearity, and partial observations. Recent advances in machine learning (ML)-based data assimilation methods have demonstrated encouraging results. In this work, we develop an ensemble score filter (EnSF) that integrates image inpainting to solve the data assimilation problems with partial observations. The EnSF method exploits an exclusively designed training-free diffusion models to solve high-dimensional nonlinear data assimilation problems. Its performance has been successfully demonstrated in the context of having full observations, i.e., all the state variables are directly or indirectly observed. However, because the EnSF does not use a covariance matrix to capture the dependence between the observed and unobserved state variables, it is nontrivial to extend the original EnSF method to the partial observation scenario. In this work, we incorporate various image inpainting techniques into the EnSF to predict the unobserved states during data assimilation. At each filtering step, we first use the diffusion model to estimate the observed states by integrating the likelihood information into the score function. Then, we use image inpainting methods to predict the unobserved state variables. We demonstrate the performance of the EnSF with inpainting by tracking the Surface Quasi-Geostrophic (SQG) model dynamics under a variety of scenarios. The successful proof of concept paves the way to more in-depth investigations on exploiting modern image inpainting techniques to advance data assimilation methodology for practical geoscience and weather forecasting problems.
Paper Structure (22 sections, 16 equations, 39 figures, 3 tables)

This paper contains 22 sections, 16 equations, 39 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Data assimilation for the SQG model
  3. The surface quasi-geostrophic (SQG) model
  4. Ensemble score filter (EnSF) with inpainting
  5. Overview of EnSF
  6. PDE-based inpainting
  7. Dictionary-learning-based inpainting
  8. Discussion on the connection between inpainitng and traditional approaches in data assimilation
  9. Numerical experiments
  10. Experimental design
  11. Comprehensive comparison
  12. LETKF fine tuning
  13. RMSE comparison
  14. Ensemble uncertainty estimation
  15. Ablation study
  16. ...and 7 more sections

Figures (39)

  • Figure 1: The proposed workflow.
  • Figure 1: Illustration of nonlinear observation using arctangent operator. Subfigure (b) demonstrates that the observation information is significantly compressed compared to the ground truth. As the percentage of available observations decreases in (c) and (d), extracting information through the nonlinear operator becomes increasingly challenging.
  • Figure 1: RMSE of $(\mathbf{C}_1)$
  • Figure 2: Illustration of PDE-based inpainting methods based on the Navier-Stokes equation and biharmonic equation for one snapshot of the SQG model. Both methods provide satisfactory reconstruction results when having 25% of the state observed. The reconstruction accuracy deteriorates when having only 5% of the state observed, which motivated us to purse dictionary-learning-based inpainting in Section \ref{['sec:DL_inpaint']}.
  • Figure 2: The LETKF fine tuning chart at the $64 \times 64$ resolution, where the case labels are defined in Table \ref{['table:scenario']}. The optimal parameter pairs, if available, are marked on the chart. An RMSE greater than 6 indicates total failure of LETKF in tracking the SQG model. The grey regions represent cases where the RMSE diverges to undefined values (i.e., NAN). There is no optimal parameter pairs for cases ($\mathbf{C}_5$) and ($\mathbf{C}_7$) due to the nonlinearity and the high observation sparsity.
  • ...and 34 more figures

Theorems & Definitions (1)

  • Remark 4.1: Reproducibility