Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PlanarNeRF: Online Learning of Planar Primitives with Neural Radiance Fields

Zheng Chen, Qingan Yan, Huangying Zhan, Changjiang Cai, Xiangyu Xu, Yuzhong Huang, Weihan Wang, Ziyue Feng, Yi Xu, Lantao Liu

TL;DR

PlanarNeRF tackles online detection of dense 3D planar primitives from monocular RGB-D sequences by integrating appearance and geometry learning in a neural radiance field. It extends NeRF with a plane rendering branch, a lightweight plane fitting module, and a global memory bank to enforce cross-frame consistency, supporting both supervised and self-supervised training. The approach demonstrates superior 3D geometry fidelity and plane instance segmentation against strong baselines while operating online with modest memory and computational demands. This yields robust, annotated-free or sparsely supervised planar reconstructions suitable for real-time robotics, AR/VR, and scene understanding.

Abstract

Identifying spatially complete planar primitives from visual data is a crucial task in computer vision. Prior methods are largely restricted to either 2D segment recovery or simplifying 3D structures, even with extensive plane annotations. We present PlanarNeRF, a novel framework capable of detecting dense 3D planes through online learning. Drawing upon the neural field representation, PlanarNeRF brings three major contributions. First, it enhances 3D plane detection with concurrent appearance and geometry knowledge. Second, a lightweight plane fitting module is proposed to estimate plane parameters. Third, a novel global memory bank structure with an update mechanism is introduced, ensuring consistent cross-frame correspondence. The flexible architecture of PlanarNeRF allows it to function in both 2D-supervised and self-supervised solutions, in each of which it can effectively learn from sparse training signals, significantly improving training efficiency. Through extensive experiments, we demonstrate the effectiveness of PlanarNeRF in various scenarios and remarkable improvement over existing works.

PlanarNeRF: Online Learning of Planar Primitives with Neural Radiance Fields

TL;DR

PlanarNeRF tackles online detection of dense 3D planar primitives from monocular RGB-D sequences by integrating appearance and geometry learning in a neural radiance field. It extends NeRF with a plane rendering branch, a lightweight plane fitting module, and a global memory bank to enforce cross-frame consistency, supporting both supervised and self-supervised training. The approach demonstrates superior 3D geometry fidelity and plane instance segmentation against strong baselines while operating online with modest memory and computational demands. This yields robust, annotated-free or sparsely supervised planar reconstructions suitable for real-time robotics, AR/VR, and scene understanding.

Abstract

Identifying spatially complete planar primitives from visual data is a crucial task in computer vision. Prior methods are largely restricted to either 2D segment recovery or simplifying 3D structures, even with extensive plane annotations. We present PlanarNeRF, a novel framework capable of detecting dense 3D planes through online learning. Drawing upon the neural field representation, PlanarNeRF brings three major contributions. First, it enhances 3D plane detection with concurrent appearance and geometry knowledge. Second, a lightweight plane fitting module is proposed to estimate plane parameters. Third, a novel global memory bank structure with an update mechanism is introduced, ensuring consistent cross-frame correspondence. The flexible architecture of PlanarNeRF allows it to function in both 2D-supervised and self-supervised solutions, in each of which it can effectively learn from sparse training signals, significantly improving training efficiency. Through extensive experiments, we demonstrate the effectiveness of PlanarNeRF in various scenarios and remarkable improvement over existing works.
Paper Structure (13 sections, 8 equations, 7 figures, 6 tables)

This paper contains 13 sections, 8 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminaries
  5. Framework Overview
  6. Plane Rendering Learning
  7. Global Memory Bank
  8. Experiments
  9. Baselines and Evaluation Metrics
  10. Datasets and Implementations
  11. Qualitative Results
  12. Ablation Studies
  13. Conclusion

Figures (7)

  • Figure 1: We introduce PlanarNeRF, a framework designed to detect dense 3D planar primitives from monocular RGB and depth sequences. The method learns plane primitives in an online fashion while drawing knowledge from both scene appearance and geometry. Displayed are outcomes from two distinct scenes (Best viewed in color). Each case exhibits two rows: the top row visualizes the reconstruction progress, while the bottom row showcases rendered 2D segmentation images at different time steps.
  • Figure 2: Overview of PlanarNeRF. PlanarNeRF processes monocular RGB and depth image sequences, enabling online pose estimation. It offers two modes: ① PlanarNeRF-S (supervised) with 2D plane annotations, and ② PlanarNeRF-SS (self-supervised) without annotations. The framework includes an efficient plane fitting module and a global memory bank for consistent plane labeling.
  • Figure 3: Qualitative comparisons of different methods for two scenes. (a) PlaneAE; (b) ESTDepth+PEAC; (c) NeuralRecon+Seq-RANSAC; (d) PlanarRecon; (e) PlanarNeRF-SS (ours); and (f) PlanarNeRF-S (ours).
  • Figure 4: Qualitative comparison between the recent SOTA --- PlanarRecon xie2022planarrecon and ours on ScanNet.
  • Figure 5: Results by PlanarNeRF for (a) Replica dataset, and (b) Synthetic dataset.
  • ...and 2 more figures