Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction

Xiao Chen, Quanyi Li, Tai Wang, Tianfan Xue, Jiangmiao Pang

TL;DR

GenNBV introduces an end-to-end, generalizable Next-Best-View policy for active 3D reconstruction by expanding the action space to five dimensions and employing a multi-source state embedding that combines a probabilistic 3D occupancy grid with semantic RGB-based features and action history. The policy, trained with PPO in a highly parallelized simulation environment, achieves strong cross-dataset generalization, delivering high coverage on unseen building-scale objects across Houses3K, OmniObject3D, and beyond. Key contributions include the free-space 5D action space, the probabilistic 3D grid occupancy representation, and the multi-source embedding that together enable robust NBV prediction without per-scene optimization. Across extensive experiments, GenNBV outperforms rule-based and prior RL-based NBV policies, demonstrating improved reconstruction completeness, accuracy, and view-efficiency on both in-distribution and cross-domain tasks, with practical implications for autonomous aerial scanning.

Abstract

While recent advances in neural radiance field enable realistic digitization for large-scale scenes, the image-capturing process is still time-consuming and labor-intensive. Previous works attempt to automate this process using the Next-Best-View (NBV) policy for active 3D reconstruction. However, the existing NBV policies heavily rely on hand-crafted criteria, limited action space, or per-scene optimized representations. These constraints limit their cross-dataset generalizability. To overcome them, we propose GenNBV, an end-to-end generalizable NBV policy. Our policy adopts a reinforcement learning (RL)-based framework and extends typical limited action space to 5D free space. It empowers our agent drone to scan from any viewpoint, and even interact with unseen geometries during training. To boost the cross-dataset generalizability, we also propose a novel multi-source state embedding, including geometric, semantic, and action representations. We establish a benchmark using the Isaac Gym simulator with the Houses3K and OmniObject3D datasets to evaluate this NBV policy. Experiments demonstrate that our policy achieves a 98.26% and 97.12% coverage ratio on unseen building-scale objects from these datasets, respectively, outperforming prior solutions.

GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction

TL;DR

GenNBV introduces an end-to-end, generalizable Next-Best-View policy for active 3D reconstruction by expanding the action space to five dimensions and employing a multi-source state embedding that combines a probabilistic 3D occupancy grid with semantic RGB-based features and action history. The policy, trained with PPO in a highly parallelized simulation environment, achieves strong cross-dataset generalization, delivering high coverage on unseen building-scale objects across Houses3K, OmniObject3D, and beyond. Key contributions include the free-space 5D action space, the probabilistic 3D grid occupancy representation, and the multi-source embedding that together enable robust NBV prediction without per-scene optimization. Across extensive experiments, GenNBV outperforms rule-based and prior RL-based NBV policies, demonstrating improved reconstruction completeness, accuracy, and view-efficiency on both in-distribution and cross-domain tasks, with practical implications for autonomous aerial scanning.

Abstract

While recent advances in neural radiance field enable realistic digitization for large-scale scenes, the image-capturing process is still time-consuming and labor-intensive. Previous works attempt to automate this process using the Next-Best-View (NBV) policy for active 3D reconstruction. However, the existing NBV policies heavily rely on hand-crafted criteria, limited action space, or per-scene optimized representations. These constraints limit their cross-dataset generalizability. To overcome them, we propose GenNBV, an end-to-end generalizable NBV policy. Our policy adopts a reinforcement learning (RL)-based framework and extends typical limited action space to 5D free space. It empowers our agent drone to scan from any viewpoint, and even interact with unseen geometries during training. To boost the cross-dataset generalizability, we also propose a novel multi-source state embedding, including geometric, semantic, and action representations. We establish a benchmark using the Isaac Gym simulator with the Houses3K and OmniObject3D datasets to evaluate this NBV policy. Experiments demonstrate that our policy achieves a 98.26% and 97.12% coverage ratio on unseen building-scale objects from these datasets, respectively, outperforming prior solutions.
Paper Structure (18 sections, 7 equations, 5 figures, 6 tables)

This paper contains 18 sections, 7 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Formulation of the Next-Best-View Problem
  5. Generalizable State Embedding
  6. Reward Function and Optimization
  7. Experiments
  8. Experimental Setup
  9. Performance Comparison
  10. Ablation Study
  11. Qualitative Results
  12. Conclusion
  13. Implementation Details
  14. Occupancy Grid Mapping Algorithm
  15. Network and Training
  16. ...and 3 more sections

Figures (5)

  • Figure 1: To determine the best view for 3D reconstruction, previous methods only chose from hand-crafted action space or based on object-centric capturing, lacking the ability to generalize to unforeseen scenes (Left). With our end-to-end trained, generalized free-space policy, it can generalize to unseen objects, enabling the captured drone to image from any viewpoint (Right).
  • Figure 2: Overview of our proposed framework GenNBV. Our end-to-end policy takes the historical multi-source observations as input, transforms them into a more informative scene representation, and produces the next viewpoint position. A reward signal will be returned at training time to optimize the end-to-end policy for maximizing the expected cumulative reward in one episode. Specifically, the signal is the increased coverage ratio after collecting a new viewpoint.
  • Figure 3: The visualization results of unseen 3D objects reconstructed by Scan-RL peralta2020next and our model to compare the generalizability. (a) Unseen buildings from the test set of Houses3K. (b) Unseen buildings from OmniObject3D. It's quite obvious that some parts of the model reconstructed by Scan-RL are wrong or missing. For example, the second object in the first row has a pillar in the wrong shape. Scan-RL fails to reconstruct the roof edge for the fourth object from OmniObject3D, as shown in the third row.
  • Figure 4: The visualization results of an unseen 3D outdoor scene with enormous details from Objaverse, reconstructed by Uncertainty-guided, Scan-RL and our model. Compared to the uncertainty-guided method and Scan-RL, the scene reconstructed by our method is more watertight and has fewer holes on the ground and building surface, especially in the region highlighted by the red box.
  • Figure 5: The curve of coverage ratio with the increasing number of training objects on unseen OmniObject3D house category.