AiSDF: Structure-aware Neural Signed Distance Fields in Indoor Scenes

TL;DR

The proposed AiSDF reconstruction framework is evaluated on the ScanNet and ReplicaCAD datasets, where it is demonstrated that the proposed framework is capable of reconstructing fine details of objects implicitly, as well as structures explicitly in room-scale scenes.

Abstract

Indoor scenes we are living in are visually homogenous or textureless, while they inherently have structural forms and provide enough structural priors for 3D scene reconstruction. Motivated by this fact, we propose a structure-aware online signed distance fields (SDF) reconstruction framework in indoor scenes, especially under the Atlanta world (AW) assumption. Thus, we dub this incremental SDF reconstruction for AW as AiSDF. Within the online framework, we infer the underlying Atlanta structure of a given scene and then estimate planar surfel regions supporting the Atlanta structure. This Atlanta-aware surfel representation provides an explicit planar map for a given scene. In addition, based on these Atlanta planar surfel regions, we adaptively sample and constrain the structural regularity in the SDF reconstruction, which enables us to improve the reconstruction quality by maintaining a high-level structure while enhancing the details of a given scene. We evaluate the proposed AiSDF on the ScanNet and ReplicaCAD datasets, where we demonstrate that the proposed framework is capable of reconstructing fine details of objects implicitly, as well as structures explicitly in room-scale scenes.

Figures (6)

• Figure 1: The proposed AiSDF on the ScanNet dataset dai2017scannet. Top: Our framework represents the scene as a signed distance fields (SDF) by considering the structure of the scene in a continual manner. Middle: In addition, we estimate the underlying Atlanta structure (global Atlanta frame) and extract a 3D explicit planar map in the form of Atlanta-aware surfels. We colorize each surfel with the associated Atlanta direction. Bottom: We visualize the last keyframe used by AiSDF (RGB image is unused in practice).
• Figure 2: Overview of AiSDF. Given a stream of posed depth images, AiSDF first selects the keyframe and adds it to the keyframe set for continual learning. We update the global Atlanta frame (AF) by extracting the dominant directions from a new keyframe and then generate surfels that represent the planar regions supported by the updated global AF. From a set of keyframes with Atlanta-aware surfels, we sample the 3D points considering the structure of the scene. Finally, sampled point x is queried to MLP that outputs signed distance value s, and we optimize the network in a self-supervised manner by measuring the loss between s and bound b. Note that we intentionally present intermediate steps of continual learning to show the process of extracting the new Atlanta direction and surfels supported by updated global AF. In Atlanta-aware sampling (blue box), we use the ground truth mesh to visualize the sampling effectively. The final mesh result indicates the reconstructed mesh by AiSDF using all keyframes.
• Figure 3: Illustration of underlying AF estimation. (a) Given global AF $\mathcal{V}_G$ (solid arrows) and surface normal distribution of new keyframe $\mathcal{K}_j$, the Atlanta structure analysis proceeds in two steps. (b) First, we estimate potential dominant horizontal directions (black bars) from a 1D histogram of inlier surface normals. (c) The new dominant direction (purple arrows) is extracted by associating potential directions with the global AF in the world coordinate.
• Figure 4: Illustration of the Atlanta-aware surfels and surfel-aware bound computation. Left: (a) Atlanta-aware surfel representation. (b) 2D surfel mask $\mathbf{M}_\mathfrak{s}$ overlaid on the depth image. Right: Selected points to compute bound values with surfels (top) and without surfels (bottom). With Atlanta-aware surfels, AiSDF can compute more tight bound values while covering a denser and wider area.
• Figure 5: Qualitative evaluation on the ScanNet and ReplicaCAD datasets. The purple boxes are close-up views to show the details of each scene. The second column presents the half-transparent explicit planar map composed of surfels overlaid on the mesh. Here, the green color denotes the surfels supported by the vertical Atlanta direction, and the other colors represent the surfels by the other horizontal Atlanta directions. Note that the size of the surfels may look smaller in the ReplicaCAD due to the different scales of scenes.