Exploiting Priors from 3D Diffusion Models for RGB-Based One-Shot View Planning

TL;DR

The paper addresses how to plan informative RGB views for reconstructing an unknown object from a single image. It introduces a pipeline that uses a 3D diffusion model to generate a mesh as a geometric prior, formulates an object-specific one-shot view planning problem as a set covering optimization with multi-view and distance constraints, and computes a globally shortest path to collect RGB views for NeRF reconstruction. Key contributions include (i) leveraging diffusion priors to enable RGB-based one-shot planning, (ii) a customized set covering formulation with alpha-view coverage and beta-distance constraints, and (iii) comprehensive simulations and real-world experiments showing improved reconstruction quality and reduced movement cost compared to baselines. The approach demonstrates the practical feasibility of diffusion-informed planning in robotics and provides open-source code to foster reproducibility and further research.

Abstract

Object reconstruction is relevant for many autonomous robotic tasks that require interaction with the environment. A key challenge in such scenarios is planning view configurations to collect informative measurements for reconstructing an initially unknown object. One-shot view planning enables efficient data collection by predicting view configurations and planning the globally shortest path connecting all views at once. However, prior knowledge about the object is required to conduct one-shot view planning. In this work, we propose a novel one-shot view planning approach that utilizes the powerful 3D generation capabilities of diffusion models as priors. By incorporating such geometric priors into our pipeline, we achieve effective one-shot view planning starting with only a single RGB image of the object to be reconstructed. Our planning experiments in simulation and real-world setups indicate that our approach balances well between object reconstruction quality and movement cost.

Figures (9)

• Figure 1: An example of our RGB-based one-shot view planning by exploiting priors from 3D diffusion models. Our goal is to plan a set of views (blue) at once to collect informative RGB images for object reconstruction. The key component in our approach is a 3D diffusion model generating the corresponding 3D mesh of a single RGB image from the initial camera view (red). By leveraging the mesh as geometric priors, our approach produces view configurations specifically associated with the target object and calculates the globally shortest path. In particular, we plan denser views to observe more geometrically complex parts (front part of the object in the example) to improve the reconstruction quality.
• Figure 2: Overview of our proposed RGB-based one-shot view planning pipeline. Given a single RGB image of the object to be reconstructed, we leverage a 3D diffusion model, One-2-3-45++ liu2023arxiv, to generate a 3D mesh. This mesh serves as a proxy to the ground truth geometry and is the basis for our view planning. Based on this prior, we construct the one-shot view planning task as a customized set covering optimization problem and solve it to obtain a minimum set of views required to densely cover the mesh surfaces. The RGB camera starts at the initial view (shown as $\otimes$) and follows the generated globally shortest path to collect RGB images, which we use to train a NeRF in Instant-NGP muller2022tog after the data acquisition is completed.
• Figure 3: Illustration of the impact of multi-view constraints. $\alpha$ denotes the minimum number of views required to observe each surface point. Larger $\alpha$ values lead to optimization solutions with more views densely covering the surfaces.
• Figure 4: Illustration of the impact of distance constraints: (a) spatially clustered views (the orange circle showcases an example of clustered views); (b) spatially more uniform views. Both view configurations are feasible solutions. By incorporating distance constraints, we express the preference for spatially uniform distribution to avoid redundant information in clustered views.
• Figure 5: Ten test objects used in our simulation experiments.