LeC$^2$O-NeRF: Learning Continuous and Compact Large-Scale Occupancy for Urban Scenes

TL;DR

This paper is the first to learn a continuous and compact occupancy of large-scale NeRF by a network, which can classify 3D points as occupied or unoccupied points, and can speed up state-of-the-art NeRF methods without sacrificing accuracy.

Abstract

In NeRF, a critical problem is to effectively estimate the occupancy to guide empty-space skipping and point sampling. Grid-based methods work well for small-scale scenes. However, on large-scale scenes, they are limited by predefined bounding boxes, grid resolutions, and high memory usage for grid updates, and thus struggle to speed up training for large-scale, irregularly bounded and complex urban scenes without sacrificing accuracy. In this paper, we propose to learn a continuous and compact large-scale occupancy network, which can classify 3D points as occupied or unoccupied points. We train this occupancy network end-to-end together with the radiance field in a self-supervised manner by three designs. First, we propose a novel imbalanced occupancy loss to regularize the occupancy network. It makes the occupancy network effectively control the ratio of unoccupied and occupied points, motivated by the prior that most of 3D scene points are unoccupied. Second, we design an imbalanced architecture containing a large scene network and a small empty space network to separately encode occupied and unoccupied points classified by the occupancy network. This imbalanced structure can effectively model the imbalanced nature of occupied and unoccupied regions. Third, we design an explicit density loss to guide the occupancy network, making the density of unoccupied points smaller. As far as we know, we are the first to learn a continuous and compact occupancy of large-scale NeRF by a network. In our experiments, our occupancy network can quickly learn more compact, accurate and smooth occupancy compared to the occupancy grid. With our learned occupancy as guidance for empty space skipping on challenging large-scale benchmarks, our method consistently obtains higher accuracy compared to the occupancy grid, and our method can speed up state-of-the-art NeRF methods without sacrificing accuracy.

Figures (9)

• Figure 1: Differences between the occupancy grid li2022nerfaccmueller2022instant and our occupancy network. Our occupancy network is a compact and continuous MLP with only 0.15M parameters, trained together with NeRF networks by our designed losses. The occupancy grid is a discrete representation and stores 2.0M and 128.0M parameters for a resolution of $128^3$ and $512^3$, respectively. It is updated by evaluating the NeRF network and is not aware of the training loss. The images are the visualization of the occupied and unoccupied points as stated in Section \ref{['sec:Metricsandvisualization']}. The whole points are sent to the grid or the occupancy network and are split into two parts of occupied and unoccupied points.
• Figure 2: Our proposed LeC$^2$O-NeRF. The occupancy network predicts the occupancy of each point and dispatches them into different sub-networks. $\textbf{x}_1$ and $\textbf{x}_2$ go through the occupancy network and are dispatched to the empty space network and the scene network, according to their occupancy values. The occupancy can be trained end-to-end together with the NeRF network by multiplying occupancy values on the output of sub-networks. If a point is dispatched into the empty space network, it is classified as unoccupied. The occupancy network is a small MLP. We enlarge the figure of the occupancy network to clearly show its operation. The imbalanced occupancy loss and the density loss are computed by the occupancy values and the detached $\sigma$.
• Figure 3: (a) The computation of the imbalanced occupancy loss and the density loss from occupancy values. (b) After training the occupancy network of a scene, we can use our frozen occupancy network to guide the sampling and training of NeRF methods.
• Figure 4: The visualization of our occupancy and the grid occupancy as point clouds. Our predicted occupied points (scene surface points) are cleaner and have fewer points than the grid occupancy. They fit the surface of the buildings more compactly.
• Figure 5: The rendered images of our occupancy network and the occupancy grid based on Switch-NeRF. Our method can obtain more complete, clean, and high-quality images.