TriNeRFLet: A Wavelet Based Triplane NeRF Representation

TL;DR

The paper addresses the limited 3D reconstruction quality of Triplane NeRF representations. It introduces TriNeRFLet, a multiscale 2D wavelet-based NeRF that learns wavelet coefficients with $L_1$ sparsity ($L_1$ regularization) across scales and employs a coarse-to-fine training regime, enabling cross-scale information flow. It further adds a diffusion-guided SR pipeline based on a pre-trained Stable Diffusion upscaler to upscale NeRF resolutions without 3D supervision. Experiments on Blender and LLFF demonstrate improved 3D recovery over Triplane and competitive super-resolution results against 2D SR baselines, validating the effectiveness of wavelet-domain feature learning for NeRF.

Abstract

In recent years, the neural radiance field (NeRF) model has gained popularity due to its ability to recover complex 3D scenes. Following its success, many approaches proposed different NeRF representations in order to further improve both runtime and performance. One such example is Triplane, in which NeRF is represented using three 2D feature planes. This enables easily using existing 2D neural networks in this framework, e.g., to generate the three planes. Despite its advantage, the triplane representation lagged behind in its 3D recovery quality compared to NeRF solutions. In this work, we propose TriNeRFLet, a 2D wavelet-based multiscale triplane representation for NeRF, which closes the 3D recovery performance gap and is competitive with current state-of-the-art methods. Building upon the triplane framework, we also propose a novel super-resolution (SR) technique that combines a diffusion model with TriNeRFLet for improving NeRF resolution.

Figures (9)

• Figure 1: Our approach improves the quality of NeRF reconstruction (zoom-in). From left to right: TriNeRFLet improvement over Triplane, TriNeRFLet compared to INGP, TriNeRFLet SR improvement, TriNeRFLet SR compared to NeRF-SR.
• Figure 1: Performance comparison between different wavelet filters and no wavelet at all (Triplane).
• Figure 2: Illustartion of the multiscale property of the wavelet representation.
• Figure 2: NeRF reconstruction qualitative results. Notice the improvement in reconstruction quality of TriNeRFLet compared to Triplane. More visual results appear in the project page.
• Figure 3: The TriNeRFLet reconstruction framework learns features in wavelet domain, which are transformed into feature domain to render 3D objects.