SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Views

TL;DR

SpaRP addresses open-world 3D reconstruction and relative pose estimation from unposed sparse-view images by finetuning a 2D diffusion model to jointly predict Normalized Object Coordinate Space (NOCS) maps for pose estimation and multi-view images for 3D reconstruction from $1\sim6$ views. It tiles sparse inputs into a conditioning grid, uses local and global conditioning, and applies a domain switcher to generate both NOCS maps and multi-view images; poses are extracted via PnP on the NOCS maps with RANSAC, and a 3D diffusion-based reconstruction yields a textured mesh, with optional pose refinement via differentiable rendering. A Mixture of Experts strategy mitigates diffusion stochasticity, improving pose accuracy for symmetric objects. The method achieves state-of-the-art performance on three datasets with significantly faster runtime (around $20$ seconds) and demonstrates strong generalization to unseen object categories, enabling rapid, controllable open-world 3D asset generation from sparse inputs.

Abstract

Open-world 3D generation has recently attracted considerable attention. While many single-image-to-3D methods have yielded visually appealing outcomes, they often lack sufficient controllability and tend to produce hallucinated regions that may not align with users' expectations. In this paper, we explore an important scenario in which the input consists of one or a few unposed 2D images of a single object, with little or no overlap. We propose a novel method, SpaRP, to reconstruct a 3D textured mesh and estimate the relative camera poses for these sparse-view images. SpaRP distills knowledge from 2D diffusion models and finetunes them to implicitly deduce the 3D spatial relationships between the sparse views. The diffusion model is trained to jointly predict surrogate representations for camera poses and multi-view images of the object under known poses, integrating all information from the input sparse views. These predictions are then leveraged to accomplish 3D reconstruction and pose estimation, and the reconstructed 3D model can be used to further refine the camera poses of input views. Through extensive experiments on three datasets, we demonstrate that our method not only significantly outperforms baseline methods in terms of 3D reconstruction quality and pose prediction accuracy but also exhibits strong efficiency. It requires only about 20 seconds to produce a textured mesh and camera poses for the input views. Project page: https://chaoxu.xyz/sparp.

Figures (9)

• Figure 1: SpaRP handles open-world 3D reconstruction and pose estimation from unposed sparse-view images, delivering results in approximately 20 seconds.
• Figure 2: Pipeline Overview of SpaRP. We begin by taking a sparse set of unposed images as input, which we tile into a single composite image. This composite image is subsequently provided to the Stable Diffusion UNet to serve as the conditioning input. The 2D diffusion model is simultaneously finetuned to predict NOCS maps for the input sparse views and multi-view images under known camera poses. From the NOCS maps, we extract the camera poses corresponding to the input views. The multi-view images are then processed by a reconstruction module to generate textured 3D meshes. Optionally, the camera poses can be further refined using the generated mesh for improved accuracy.
• Figure 3: (a) Regardless of the poses of the sparse input views (in black), the output multiviews are uniformly distributed (in red) and encompass the entire 3D object. (b) The Normalized Object Coordinate Space (NOCS) of the object, whose orientation is aligned with the azimuth of the first input view. (c) An example of input and output tiled images. The elevation and azimuth of the first input view are denoted by $\theta_0$ and $\phi_0$, respectively. The camera poses of the output multiview images are determined by $\phi_0$. The output NOCS maps correspond to the input sparse views, and the orientation of the coordinate frame is also determined by $\phi_0$.
• Figure 4: Qualitative Results on 3D Reconstruction. Zero123XL deitke2023objaversexl, One2345 liu2023one2345, and TripoSR tochilkin2024triposr are single-image-to-3D methods, each utilizing only the first input image. iFusion wu2023ifusion, EscherNet kong2024eschernet, and our approach take all input images (the first row). Textured meshes and mesh normal renderings are shown. Shapes come from the OmniObject3D wu2023omniobject3d and GSO downs2022google datasets.
• Figure 5: Single-View vs. Sparse-View for 3D Reconstruction. We compare the results of our method when using single-view and sparse-view inputs.