Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields

TL;DR

Mip-NeRF 360 extends mip-NeRF to unbounded, 360-degree scenes by introducing a Kalman-like contraction to parameterize space, a coarse-to-fine online distillation framework with a small proposal MLP and a large NeRF MLP, and a distortion-based regularizer to control interval-based representations. The method co-designs scene parameterization with disparity-based ray sampling and achieves a 57% reduction in MSE over mip-NeRF while maintaining practical training costs. It delivers realistic novel views and high-detail depth maps on challenging real-world scenes, without relying on proxy geometry. These improvements advance neural radiance field rendering for unconstrained camera paths and distant content, expanding applicability to large-scale, real-world environments.

Abstract

Though neural radiance fields (NeRF) have demonstrated impressive view synthesis results on objects and small bounded regions of space, they struggle on "unbounded" scenes, where the camera may point in any direction and content may exist at any distance. In this setting, existing NeRF-like models often produce blurry or low-resolution renderings (due to the unbalanced detail and scale of nearby and distant objects), are slow to train, and may exhibit artifacts due to the inherent ambiguity of the task of reconstructing a large scene from a small set of images. We present an extension of mip-NeRF (a NeRF variant that addresses sampling and aliasing) that uses a non-linear scene parameterization, online distillation, and a novel distortion-based regularizer to overcome the challenges presented by unbounded scenes. Our model, which we dub "mip-NeRF 360" as we target scenes in which the camera rotates 360 degrees around a point, reduces mean-squared error by 57% compared to mip-NeRF, and is able to produce realistic synthesized views and detailed depth maps for highly intricate, unbounded real-world scenes.

Figures (13)

• Figure 1: (a) Though mip-NeRF is able to produce accurate renderings of objects, for unbounded scenes it often generates blurry backgrounds and low-detail foregrounds. (b) Our model produces detailed realistic renderings of these unbounded scenes, as evidenced by the renderings (top) and depth maps (bottom) from both models. See the supplemental video for additional results.
• Figure 2: A 2D visualization of our scene parameterization. We define a $\operatorname{contract}(\cdot)$ operator (Equation \ref{['eq:contract']}, shown as arrows) that maps coordinates onto a ball of radius 2 (orange), where points within a radius of 1 (blue) are unaffected. We apply this contraction to mip-NeRF Gaussians in Euclidean 3D space (gray ellipses) similarly to a Kalman filter to produce our contracted Gaussians (red ellipses), whose centers are guaranteed to lie within a ball of radius 2. The design of $\operatorname{contract}(\cdot)$ combined with our choice to space ray intervals linearly according to disparity means that rays cast from a camera located at the origin of the scene will have equidistant intervals in the orange region, as demonstrated here.
• Figure 3: A comparison of our model's architecture with mip-NeRF's. Mip-NeRF uses one multi-scale MLP that is repeatedly queried (only two repetitions shown here) for weights that are resampled into intervals for the next stage, and supervises the renderings produced at all scales. We use a "proposal MLP" that emits weights (but not color) that are resampled, and in the final stage we use a "NeRF MLP" to produce weights and colors that result in the rendered image, which we supervise. The proposal MLP is trained to produce proposal weights $\hat{\mathbf{w}}$ that are consistent with the NeRF MLP's $\mathbf{w}$ output. By using a small proposal MLP and a large NeRF MLP we obtain a combined model with a high capacity that is still tractable to train.
• Figure 4: A visualization of the histograms $(\mathbf{t}, \mathbf{w})$ emitted from the NeRF MLP (black) and the two sets of histograms $(\hat{\mathbf{t}}, \hat{\mathbf{w}})$ emitted by the proposal MLP (yellow and orange) for a single ray from our dataset's bicycle scene over the course of training. Below we visualize the entire ray with fixed x and y axes, but above we crop both axes to better visualize details near scene content. Histogram weights are plotted as distributions that integrate to $1$. (a) When training begins, all weights are uniformly distributed with respect to ray distance $t$. (b, c) As training progresses, the NeRF weights begin to concentrate around a surface and the proposal weights form a kind of envelope around those NeRF weights.
• Figure 5: Our regularizer suppresses "floaters" (pieces of semi-transparent material floating in space, which are easy to identify in the depth map) and prevents a phenomenon in which surfaces in the background "collapse" towards the camera (shown in the bottom left of (a)). The noise-injection approach of Mildenhall et al. mildenhall2020 only partially eliminates these artifacts, and reduces reconstruction quality (note the lack of detail in the depths of the distant trees). See the supplemental video for more visualizations.