Multi-Fidelity Residual Neural Processes for Scalable Surrogate Modeling

TL;DR

Multi-fidelity Residual Neural Processes (MFRNP) explicitly models the residual between the aggregated output from lower fidelities and ground truth at the highest fidelity, and significantly outperforms state-of-the-art in learning partial differential equations and a real-world climate modeling task.

Abstract

Multi-fidelity surrogate modeling aims to learn an accurate surrogate at the highest fidelity level by combining data from multiple sources. Traditional methods relying on Gaussian processes can hardly scale to high-dimensional data. Deep learning approaches utilize neural network based encoders and decoders to improve scalability. These approaches share encoded representations across fidelities without including corresponding decoder parameters. This hinders inference performance, especially in out-of-distribution scenarios when the highest fidelity data has limited domain coverage. To address these limitations, we propose Multi-fidelity Residual Neural Processes (MFRNP), a novel multi-fidelity surrogate modeling framework. MFRNP explicitly models the residual between the aggregated output from lower fidelities and ground truth at the highest fidelity. The aggregation introduces decoders into the information sharing step and optimizes lower fidelity decoders to accurately capture both in-fidelity and cross-fidelity information. We show that MFRNP significantly outperforms state-of-the-art in learning partial differential equations and a real-world climate modeling task. Our code is published at: https://github.com/Rose-STL-Lab/MFRNP

Figures (3)

• Figure 1: Graphical model comparison of MFRNP against the state-of-the-art D-MFD baseline model. Shaded circles denote observed variables. Right: D-MFD disentangles the latent representations $r_{k,n}$ into local and global representations $L_{k,n}$ and $G_{k,n}$ to infer $z_k$. Each $G_{k,n}$ is from a different fidelity level, while no information about the decoder parameters $\theta_k$ is shared, making $G_{k,n}$ inaccurate regarding $\theta_i$ where $i\neq k$. Left: MFRNP dynamically constructs $\mathcal{D}^{train}_K = \{X_K, R(X_K)\}$ in cross-fidelity optimization step and learn fidelity-specific information with in-fidelity learning step. The residual function $R$ makes $z_K$ dependant on $z_{1:K-1}$ and $\theta_{1:K-1}$ without inflating any dimensions. Thus, MFRNP optimizes lower fidelity decoders for better information sharing. We use bold letters in our model graph to denote a set of variables.
• Figure 2: Perfect model test performance for climate modeling task. Measured in latitude-weighted nRMSE. MFRNP outperforms other models in 3 out of the 4 scenarios and maintains consistent performance across all the scenarios from year $2015\sim2100$.
• Figure 3: Absolute residual plot (°C) between ground truth and model predictions on ERA5-reanalysis data at year $2020$, with global resolution of $721\times1440$. The first row presents global view, second row focuses on smaller geographic regions. The third row shows the latitude weighted nRMSE to measure prediction accuracy from year 2015 to 2021. MFRNP outperforms other models staring from $2017$.