Boost Your NeRF: A Model-Agnostic Mixture of Experts Framework for High Quality and Efficient Rendering

TL;DR

This study introduces a model-agnostic framework inspired by Sparsely-Gated Mixture of Experts to enhance rendering quality without escalating computational complexity and proposes a resolution-based routing technique to effectively induce sparsity and decompose scenes.

Abstract

Since the introduction of NeRFs, considerable attention has been focused on improving their training and inference times, leading to the development of Fast-NeRFs models. Despite demonstrating impressive rendering speed and quality, the rapid convergence of such models poses challenges for further improving reconstruction quality. Common strategies to improve rendering quality involves augmenting model parameters or increasing the number of sampled points. However, these computationally intensive approaches encounter limitations in achieving significant quality enhancements. This study introduces a model-agnostic framework inspired by Sparsely-Gated Mixture of Experts to enhance rendering quality without escalating computational complexity. Our approach enables specialization in rendering different scene components by employing a mixture of experts with varying resolutions. We present a novel gate formulation designed to maximize expert capabilities and propose a resolution-based routing technique to effectively induce sparsity and decompose scenes. Our work significantly improves reconstruction quality while maintaining competitive performance.

Figures (15)

• Figure 1: Different strategies for improving reconstruction quality with Fast-NeRFs (DVGO sun2022direct). Increasing the resolution of data structures like voxel grids can improve render quality, but only up to a certain point, after which quality declines (left). The MLP component's impact on rendering quality is analyzed by varying its depth and width while keeping other variables constant (center). The effect of the number of sampled points along each ray is examined by decreasing the step size (right). In gray, our method's performance, which significantly improves PSNR at a low computational cost.
• Figure 2: A density field is used to compute density values for sampled points along a ray. A filtering step discards low-density points, routed through gating network $G$ for expert assignment. Top-$K$ experts compute radiance and density, aggregating and weighting these values by the corresponding gating probability to get the final values $\mathbf{c}_f$ and $\sigma_f$. Volume rendering equation yields pixel colors, and joint optimization refines our resolution-weighted auxiliary loss, allowing for high-quality and efficient rendering.
• Figure 2: Comparison of different gate resolutions and formulations on Lego scene and DVGO.
• Figure 3: PSNR/GFLOPs plots for Synthetic-NeRF and TanksAndTemple. The remaining datasets show similar results.
• Figure 4: Qualitative results on some of the scenes of each dataset. From left to right: ground truths, baselines, Top-1, Top-2 and Ensemble. Each model has the same parameters and has been trained for the same number of iterations (DVGO).