CUPID: Generative 3D Reconstruction via Joint Object and Pose Modeling

TL;DR

Cupid addresses the challenge of reconstructing 3D objects from a single image with unknown viewpoint by jointly modeling the object and camera pose in a probabilistic framework. It introduces a two-stage cascaded flow model: first, occupancy and UV-pose generation provides a robust pose estimate via PnP; second, a pose-aligned refinement injects pixel-level cues to recover accurate geometry and appearance. The approach achieves state-of-the-art monocular geometry fidelity (outperforming existing methods by over 3 dB PSNR and reducing Chamfer Distance) and extends naturally to multi-view and scene-level reconstruction without post-hoc optimization. By unifying 3D generation priors with explicit pose reasoning, Cupid enables faithful reconstructions and scalable scene composition for single-view and beyond.

Abstract

We introduce Cupid, a generative 3D reconstruction framework that jointly models the full distribution over both canonical objects and camera poses. Our two-stage flow-based model first generates a coarse 3D structure and 2D-3D correspondences to estimate the camera pose robustly. Conditioned on this pose, a refinement stage injects pixel-aligned image features directly into the generative process, marrying the rich prior of a generative model with the geometric fidelity of reconstruction. This strategy achieves exceptional faithfulness, outperforming state-of-the-art reconstruction methods by over 3 dB PSNR and 10% in Chamfer Distance. As a unified generative model that decouples the object and camera pose, Cupid naturally extends to multi-view and scene-level reconstruction tasks without requiring post-hoc optimization or fine-tuning.

Figures (10)

• Figure 1: Generative 3D reconstruction from a single image. Left: Input image. Middle: Our method recovers a canonical 3D model (intrinsic properties such as shape and texture) along with the exact camera pose (extrinsic property) that generated the input. Right: Re-rendering the 3D model from the estimated pose faithfully reproduces the input.
• Figure 2: Results for generative 3D reconstruction from a single test image. Given an input image (top left), Cupid estimates camera pose (bottom left) and reconstructs 3D model (bottom right), re-rendering the input (top right). It is robust to changes in scale, placement, and lighting while preserving fine details, and supports component-aligned scene reconstruction (bottom row). All results are produced in seconds via feed-forward sampling of the learned model. See https://cupid3d.github.io/ for an immersive view of the interactive 3D results.
• Figure 3: Overview of Cupid's pipeline. From a single input image, Cupid first generates an occupancy and a UV cube in canonical space. Then, a Perspective-n-Point (PnP) solver (i.e., Eq. \ref{['eq:pnp']}) recovers the camera pose. Using this recovered camera pose, we extract pose-aligned conditioning latents and visual features (i.e., Eq. \ref{['eq:latent']}), along with noisy structured latents, to generate the geometry and appearance features, which will be decoded to the 3D Gaussian splats and mesh.
• Figure 4: Component-aligned scene reconstruction. For a scene with multiple objects, our method can rebuild each object using the occlusion-aware 3D generator and then solve 3D–3D similarity transformation to accurately recompose the scene.
• Figure 5: Qualitative comparison on input view consistency. We render the input view using its generated camera pose. For view centric methods (LRM, LaRa), we use ground-truth intrinsic for rendering as they do not model intrinsic. Our method produces the highest-fidelity geometry and appearance; LRM hallucinates incorrect details, LaRa is overly blurry due to 2D diffusion inconsistencies, and 3D generation method OnePoseGen frequently fails to register pose reliably.