Flow Distillation Sampling: Regularizing 3D Gaussians with Pre-trained Matching Priors

TL;DR

This work tackles the geometry degradation of 3D Gaussian Splatting (3DGS) in sparse-view scenarios by introducing Flow Distillation Sampling (FDS), which distills a pretrained optical-flow matching prior into the 3DGS training loop. FDS couples Radiance Flow, derived from the current Gaussian geometry, with Prior Flow from a pretrained model via a dedicated loss $L_{fds}$ and a depth-aware camera sampling scheme that generates informative unobserved views. The approach yields substantial improvements in depth rendering, novel-view synthesis, and mesh quality across MushRoom, ScanNet V2, and Replica datasets, while enabling mutual refinement between the two flow maps and preserving training stability. These results highlight the potential of using metric-depth-enforcing priors from optical flow to regularize neural radiance fields, improving accuracy and realism in challenging indoor scenes and beyond.

Abstract

3D Gaussian Splatting (3DGS) has achieved excellent rendering quality with fast training and rendering speed. However, its optimization process lacks explicit geometric constraints, leading to suboptimal geometric reconstruction in regions with sparse or no observational input views. In this work, we try to mitigate the issue by incorporating a pre-trained matching prior to the 3DGS optimization process. We introduce Flow Distillation Sampling (FDS), a technique that leverages pre-trained geometric knowledge to bolster the accuracy of the Gaussian radiance field. Our method employs a strategic sampling technique to target unobserved views adjacent to the input views, utilizing the optical flow calculated from the matching model (Prior Flow) to guide the flow analytically calculated from the 3DGS geometry (Radiance Flow). Comprehensive experiments in depth rendering, mesh reconstruction, and novel view synthesis showcase the significant advantages of FDS over state-of-the-art methods. Additionally, our interpretive experiments and analysis aim to shed light on the effects of FDS on geometric accuracy and rendering quality, potentially providing readers with insights into its performance. Project page: https://nju-3dv.github.io/projects/fds

Figures (5)

• Figure 1: Pipeline of the proposed FDS. For each input view, we apply the FDS camera sampling scheme to generate corresponding unobserved sampled view. We then compute Radiance flow base on rendered depth and the Prior flow from matching prior model. Finally the Prior Flow is used to supervise Radiance flow, which enhances the geometric quality of Gaussian Radiance Field.
• Figure 2: Explanation of depth-adaptive translation radius. A fixed-radius camera sampling scheme may result in significantly different flow values (Flow 1 and Flow 2) in areas with varying depth ($d_1$ and $d_2$).
• Figure 3: Comparison of depth reconstruction on Mushroom and ScanNet datasets. The original 3DGS or 2DGS model equipped with FDS can remove unwanted floaters and reconstruct geometry more preciously.
• Figure 4: The error map of Radiance Flow and Prior Flow. RF: Radiance Flow, PF: Prior Flow, * means that there is no FDS loss supervision during optimization.
• Figure 5: Limitation of FDS.