RoGUENeRF: A Robust Geometry-Consistent Universal Enhancer for NeRF

TL;DR

RoGUENeRF introduces a geometry-consistent NeRF enhancer that fuses 3D alignment, non-rigid refinement, and geometry-aware attention with a pre-trained 2D enhancer to restore high-frequency detail while preserving 3D consistency. By selecting nearby views, reprojecting features into a novel view, refining alignments with optical flow, and regulating contributions via geometry-aware attention, it achieves robust improvements across six baselines on LLFF, 360v2, and DTU, including resilience to camera pose errors. Quantitatively, it yields gains in $PSNR$, $SSIM$, and $LPIPS$, and qualitative results show sharper high-frequency textures without flickering. The method enables practical, scalable per-scene enhancement with fine-tuning in under 60 minutes, advancing real-world neural rendering where geometry and calibration imperfections are common.

Abstract

Recent advances in neural rendering have enabled highly photorealistic 3D scene reconstruction and novel view synthesis. Despite this progress, current state-of-the-art methods struggle to reconstruct high frequency detail, due to factors such as a low-frequency bias of radiance fields and inaccurate camera calibration. One approach to mitigate this issue is to enhance images post-rendering. 2D enhancers can be pre-trained to recover some detail but are agnostic to scene geometry and do not easily generalize to new distributions of image degradation. Conversely, existing 3D enhancers are able to transfer detail from nearby training images in a generalizable manner, but suffer from inaccurate camera calibration and can propagate errors from the geometry into rendered images. We propose a neural rendering enhancer, RoGUENeRF, which exploits the best of both paradigms. Our method is pre-trained to learn a general enhancer while also leveraging information from nearby training images via robust 3D alignment and geometry-aware fusion. Our approach restores high-frequency textures while maintaining geometric consistency and is also robust to inaccurate camera calibration. We show that RoGUENeRF substantially enhances the rendering quality of a wide range of neural rendering baselines, e.g. improving the PSNR of MipNeRF360 by 0.63dB and Nerfacto by 1.34dB on the real world 360v2 dataset.

Figures (5)

• Figure 1: Novel views from the MipNeRF360 dataset barron22. RoGUENeRF achieves noticeable qualitative improvements over state-of-the-art baselines and NeRF enhancers, especially in high-frequency regions such as trees, buildings and text.
• Figure 2: RoGUENeRF Overview: Given a trained NeRF model and corresponding training data, our method substantially enhances the rendering quality of the NeRF while maintaining view-consistency.
• Figure 3: Qualitative comparisons with six different NeRF baseline models across three datasets. Our method recovers more detail in high-frequency regions such as foliage, tarmac floor, patterns on the glove, the edges of the building and text.
• Figure 4: Qualitative results when adding medium pose noise. The quality of the baseline suffers greatly while our enhancer is robust and achieves noticeable improvements.
• Figure 5: Qualitative results of the ablation study. Settings correspond to Table \ref{['table:ablation-all']}.