Neuroexplicit Diffusion Models for Inpainting of Optical Flow Fields

TL;DR

The paper tackles inpainting sparse optical flow fields by a neuroexplicit diffusion model that learns a diffusion tensor and per-pixel discretization parameters, embedded in a coarse-to-fine diffusion framework. By merging explicit PDE-based regularization with neural parameter estimation, it achieves strong reconstruction quality, data efficiency, and generalization, outperforming both fully explicit and fully data-driven baselines on Sintel and KITTI benchmarks. The approach uses a Diffusion Tensor Module to predict diffusion dynamics from the reference image and a learned Perona–Malik diffusivity, enabling adaptive, edge-aware inpainting with fewer learnable parameters and competitive runtimes. This work demonstrates that integrating explicit mathematical structure with learned components can yield interpretable, robust diffusion-based regularization suitable for practical optical flow applications such as autonomous driving.

Abstract

Deep learning has revolutionized the field of computer vision by introducing large scale neural networks with millions of parameters. Training these networks requires massive datasets and leads to intransparent models that can fail to generalize. At the other extreme, models designed from partial differential equations (PDEs) embed specialized domain knowledge into mathematical equations and usually rely on few manually chosen hyperparameters. This makes them transparent by construction and if designed and calibrated carefully, they can generalize well to unseen scenarios. In this paper, we show how to bring model- and data-driven approaches together by combining the explicit PDE-based approaches with convolutional neural networks to obtain the best of both worlds. We illustrate a joint architecture for the task of inpainting optical flow fields and show that the combination of model- and data-driven modeling leads to an effective architecture. Our model outperforms both fully explicit and fully data-driven baselines in terms of reconstruction quality, robustness and amount of required training data. Averaging the endpoint error across different mask densities, our method outperforms the explicit baselines by 11-27%, the GAN baseline by 47% and the Probabilisitic Diffusion baseline by 42%. With that, our method sets a new state of the art for inpainting of optical flow fields from random masks.

Figures (5)

• Figure 1: We present a neuroexplicit architecture for inpainting of optical flow fields. An explicit inpainting model based on partial differential equations (PDEs) is enhanced with neural parameter selection. The resulting model inherits both the exceptional out-of-domain generalization from the explicit side and the reconstruction quality from the neural side.
• Figure 2: Our proposed hybrid inpainting model. The Diffusion Tensor Module takes the reference image as input, and outputs a specific diffusion tensor $\bm{D}$ and discretization parameter $\alpha$ for every stage of the coarse-to-fine inpainting pipeline. The inpainting itself is done using a stable and well-posed anisotropic diffusion process that solves $t$ steps of the explicit scheme in Equation \ref{['explicitStep']}.
• Figure 3: Our proposed method is robust to changes in the training and inference setting. The left plot shows the weaker reliance of our method on training data. Using $194$ samples, we reach a competitive performance to the network trained on the full dataset. The right plot shows the favorable generalization to unseen mask densities of our method and the explicit EED inpainting. As indicated by the gray vertical line, we evaluated each model optimized for $5\%$ on previously unseen mask densities.
• Figure 4: Weights and inference time of the models. Compared to the baselines, our model is very lightweight and has competitive inference times. Notably, we omitted the explicit baselines, since there is no clear way to compare the methods.
• Figure 5: Samples generated with a mask density of $5\%$. Every other row displays the zoomed in area of the red rectangle in the row above. Our method manages to retain a much higher level of detail in the reconstructed flow fields. In the bottom row at the dragons chin, we can observe that the PDE methods (EED, AMLE, and LB) fail to maintain flow edges in low contrast regions. Notably, both WGAIN and the PD model have poor out-of-distribution performance. WGAIN tends to have large outliers and fails in the zero flow in the background. The PD model fails to reproduce the fuzzy material of the shamans beard due to a lack of comparable materials in the training data.