MonoSelfRecon: Purely Self-Supervised Explicit Generalizable 3D Reconstruction of Indoor Scenes from Monocular RGB Views

TL;DR

MonoSelfRecon tackles the challenge of explicit 3D indoor reconstruction from monocular RGB in a fully self-supervised, generalizable framework. It jointly learns voxel-SDF and a generalizable NeRF within a coarse-to-fine autoencoder, guided by self-supervised losses that connect SDF inputs, NeRF inputs, and their interactions. The approach delivers explicit 3D meshes that rival supervised voxel-SDF methods while outperforming existing self-supervised depth estimators, and it supports few-shot transfer to new indoor domains. Its contributions include Attentional View Fusion, GRU-based fragment fusion, and a plane-aware co-planar loss, all enabling robust, scalable reconstruction without depth or SDF annotations. Practically, this facilitates rapid, scalable indoor 3D mesh generation from monocular sequences with strong generalization capabilities.

Abstract

Current monocular 3D scene reconstruction (3DR) works are either fully-supervised, or not generalizable, or implicit in 3D representation. We propose a novel framework - MonoSelfRecon that for the first time achieves explicit 3D mesh reconstruction for generalizable indoor scenes with monocular RGB views by purely self-supervision on voxel-SDF (signed distance function). MonoSelfRecon follows an Autoencoder-based architecture, decodes voxel-SDF and a generalizable Neural Radiance Field (NeRF), which is used to guide voxel-SDF in self-supervision. We propose novel self-supervised losses, which not only support pure self-supervision, but can be used together with supervised signals to further boost supervised training. Our experiments show that "MonoSelfRecon" trained in pure self-supervision outperforms current best self-supervised indoor depth estimation models and is comparable to 3DR models trained in fully supervision with depth annotations. MonoSelfRecon is not restricted by specific model design, which can be used to any models with voxel-SDF for purely self-supervised manner.

Figures (8)

• Figure 1: MonoSelfRecon Pipeline. We use monocular RGB sequence to estimate explicit 3D mesh of generalizble indoor scenes under purely self-supervision free of SDF or depth annotations. A coarse-to-fine architecture is used for both SDF and NeRF decoders, where our proposed self-supervised losses are used between SDF-inputRGB, NeRF-inputRGB, and SDF-NeRF.
• Figure 2: Self-supervised SDF photometric loss between source and target views. Left: Voxel center is inside of the surface, SDF is negative. Orange arrows show projecting rays from voxel centers to 2D pixels (P1, P2) on each camera plane, blue arrows show reprojection of surface 3D points (S1, S2) to 2D pixels (P1', P2') in each camera plane. Surface points are estimated by SDF estimation. Right: Voxel center is outside of the surface, SDF is positive. The loss is extended to all n views in a fragment.
• Figure 3: Visual Results on ScanNet. 3D meshes are shown at top, 2D rendered depth maps are shown at bottom. Our self-supervised results are clearly better than SOTA self-supervised methods on challenging cases and fuzzy RGB input, even better than supervised depth estimation for a few cases. With weak supervision, our result is clearly better than NeuralRecon with weak supervision, which demonstrates our self-supervised design.
• Figure 4: Inter/Intra-fragment losses illustration.
• Figure 5: Multi-Plane Image (MPI) NeRF illustration.