One-Shot Refiner: Boosting Feed-forward Novel View Synthesis via One-Step Diffusion

TL;DR

The paper tackles sparse-view novel view synthesis (NVS) by addressing the efficiency and 3D-structure inconsistencies of ViT-based backbones and 2D diffusion refiners. It proposes a cohesive architecture that fuses a Geometry Transformer Backbone with a Detail-Aware Scene Reconstruction module and a one-step Feature-Guided Diffusion refinement, producing 3D-consistent, high-frequency renderings. Gaussian primitives are augmented with rich features and a structured diffusion objective combines reconstruction and perceptual guidance, with the training objective $L_{total}= \lambda_r L_r + \lambda_d L_d + \lambda_g L_g$ guiding end-to-end optimization. On DL3DV and RE10K datasets, the approach yields superior NVS quality and 3D consistency over state-of-the-art feed-forward methods, enabling fast, high-fidelity novel view synthesis from unposed sparse inputs.

Abstract

We present a novel framework for high-fidelity novel view synthesis (NVS) from sparse images, addressing key limitations in recent feed-forward 3D Gaussian Splatting (3DGS) methods built on Vision Transformer (ViT) backbones. While ViT-based pipelines offer strong geometric priors, they are often constrained by low-resolution inputs due to computational costs. Moreover, existing generative enhancement methods tend to be 3D-agnostic, resulting in inconsistent structures across views, especially in unseen regions. To overcome these challenges, we design a Dual-Domain Detail Perception Module, which enables handling high-resolution images without being limited by the ViT backbone, and endows Gaussians with additional features to store high-frequency details. We develop a feature-guided diffusion network, which can preserve high-frequency details during the restoration process. We introduce a unified training strategy that enables joint optimization of the ViT-based geometric backbone and the diffusion-based refinement module. Experiments demonstrate that our method can maintain superior generation quality across multiple datasets.

Figures (5)

• Figure 1: Starting from unposed input images, our method reconstructs 3D Gaussians within a canonical space, and leverages a one-step Stable Diffusion (SD) module to synthesize high-fidelity target views.
• Figure 2: Overview of our pipeline. Starting from a set of unposed images, we first perform spatial downsampling and feed them into a Vision Transformer (ViT)-based backbone for global feature extraction. Simultaneously, we employ a Dual-Domain Detail Perception Module to enhance fine-grained detail perception from both spatial and frequency domains. The fused features are passed into Gaussian Parameter Prediction Heads to directly predict Gaussians with features in a canonical space. Finally, a Single-step Denoising module refines these outputs to produce higher-quality novel view synthesis (NVS) results.
• Figure 3: Structure of the diffusion model in NVS.
• Figure 4: Qualitative comparison on DL3DV and RE10K datasets, all evaluated at a uniform resolution of $\mathbf{512 \times 512}$. Compared with other methods, our approach is capable of recovering finer texture details.
• Figure 5: Qualitative ablation results validating the effectiveness of each component. Incorporating Dual-Domain Detail Perception Module (DD-DPM) and the feature-guided SD refine module yields substantially higher-quality novel view synthesis.