FLINT: Learning-based Flow Estimation and Temporal Interpolation for Scientific Ensemble Visualization

TL;DR

FLINT addresses the challenge of reconstructing meaningful flow fields and temporally interpolating scalar fields in scientific ensembles where flow data may be incomplete or unavailable. It introduces a learning-based pipeline built on a four-block CNN with an online teacher-student distillation mechanism, capable of producing both flow fields and high-quality interpolants for 2D+time and 3D+time data. The method supports flow-supervised and flow-unsupervised training, leveraging forward/backward warping and a learned fusion mask to merge intermediate results, and it outperforms state-of-the-art baselines on multiple datasets while offering faster inference. Practically, FLINT enables enriched visualization and analysis by providing actionable flow information alongside interpolated densities, with demonstrated utility on simulations and experiments and potential for broad domain impact including cosmology and fluid dynamics.

Abstract

We present FLINT (learning-based FLow estimation and temporal INTerpolation), a novel deep learning-based approach to estimate flow fields for 2D+time and 3D+time scientific ensemble data. FLINT can flexibly handle different types of scenarios with (1) a flow field being partially available for some members (e.g., omitted due to space constraints) or (2) no flow field being available at all (e.g., because it could not be acquired during an experiment). The design of our architecture allows to flexibly cater to both cases simply by adapting our modular loss functions, effectively treating the different scenarios as flow-supervised and flow-unsupervised problems, respectively (with respect to the presence or absence of ground-truth flow). To the best of our knowledge, FLINT is the first approach to perform flow estimation from scientific ensembles, generating a corresponding flow field for each discrete timestep, even in the absence of original flow information. Additionally, FLINT produces high-quality temporal interpolants between scalar fields. FLINT employs several neural blocks, each featuring several convolutional and deconvolutional layers. We demonstrate performance and accuracy for different usage scenarios with scientific ensembles from both simulations and experiments.

Figures (21)

• Figure 1: Overview of FLINTpipeline during inference. The trained deep neural network performs flow field estimation$\hat{F}_{t}$ and temporal (scalar) field interpolation $\hat{D}_{t}$, where $s < t < u$, by utilizing the available densities $D_s$ and $D_{u}$ from the previous and following timesteps.
• Figure 2: FLINTnetwork architecture and pipeline during training (see corresponding Fig. \ref{['fig:nn_overview']} for inference): given the input fields $D_s$, $D_{u}$, $D^{\hbox{\scalefont{0.6}{\it{GT}}}}_t$ (GT scalar field at time $t$) and $F^{\hbox{\scalefont{0.6}{\it{GT}}}}_t$ (GT flow at time $t$) in scenarios in which the latter is available, FLINTpredicts the $\hat{D}_{t}$ scalar field and $\hat{F}_{t}$$\hat{F}^{i}_{t}$ flow fields in the outputused in the loss function for optimizing network parameters. The FLINTmodel architecture and loss function are shown in the orange box. The model consists of several stacked blocks of the convolutional network, which takes $D_s$, $D_{u}$, and $t$ as input and outputs estimated flows $\hat{F}_{t \rightarrow s}$, $\hat{F}_{t \rightarrow u}$, and the fusion mask $M$in the $i^{th}$$\hbox{Conv Block$^{}$}$ computes estimated flows ${\hat{F}^{i}_{t\rightarrow s}}, {\hat{F}^{i}_{t\rightarrow u}}$, and fusion mask $M^i$ used for interpolation. We obtain the best results with four ($N=4$) blocks. The teacher block Conv Block$^{teach}$, which receives $D^{\hbox{\scalefont{0.6}{\it{GT}}}}_t$ (the GT field at time $t$, see red arrows) as additional input, is only used during training. The zoomed-in view highlights the structure of a generic Conv Blockconsisting of backward warping, concatenation, as well as convolutional and deconvolutional layers with specified strides ($st.$). The $D^{\hbox{\scalefont{0.6}{\it{GT}}}}_t$ input at the concatenation stage is only used for the teacher Conv Block.The loss function, which can be adjusted depending on the scenario, is shown on the right within the orange box. $W_i$ and $W_i^{teach}$ represent the $i^{th}$ weight matrix of the last convolutional block in the student and teacher network, respectively. FLINTuses the same model architecture for ensembles with and without available velocityGT flow fields. The GT flow $F^{\hbox{\scalefont{0.6}{\it{GT}}}}_t$ is only used in the loss function $\mathcal{L}_{flow}$.
• Figure 3: Illustration of backward warping $\hbox{$\mathbin{W \hbox{$\mathbin{^\leftarrow}$}}$}$: (scalar) fields $D_s$ and $D_u$ are reversely mapped according to the flow fields $\hat{F}_{t \rightarrow s}$ and $\hat{F}_{t \rightarrow u}$. The fields ${\hat{D}^{}_{t\leftarrow s}}$ and ${\hat{D}^{}_{t\leftarrow u}}$ are then reconstructed using bilinear interpolation considering the values at the coordinates shown in green.
• Figure 4: LBS ensemble: FLINTflow field estimation and temporal density interpolation at the timesteps with an asterisk (*)---(a) 6$\times$ and (b) 18$\times$ interpolation. Each subfigure, from top to bottom shows GT density, FLINTinterpolated density, difference with GT density (magnified by $\times 100$), RIFE interpolated density, difference with GT density (magnified by $\times 100$); GT flow, FLINTflow estimation, difference with GT flow, flow estimated by RIFE, difference with GT flow, GT pathlines, and FLINT pathlines. The colorbar on the top right maps density, and the one on the bottom right maps flow magnitude.
• Figure 5: Droplets ensemble: FLINTflow field estimationand temporal density interpolation during inference---at the timesteps with an asterisk (*)---3$\times$ interpolation. From top to bottom, the rows show GT density, FLINTinterpolated density, difference to GT density (magnified by $\times 5$), RIFE interpolated density, difference to GT density (magnified by $\times 5$), FLINTflow estimation in HSV (see bottom right), RIFE flow estimation in HSV, flow glyphs for FLINT, and flow glyphs for RIFE.