UniRecGen: Unifying Multi-View 3D Reconstruction and Generation

Abstract

Sparse-view 3D modeling represents a fundamental tension between reconstruction fidelity and generative plausibility. While feed-forward reconstruction excels in efficiency and input alignment, it often lacks the global priors needed for structural completeness. Conversely, diffusion-based generation provides rich geometric details but struggles with multi-view consistency. We present UniRecGen, a unified framework that integrates these two paradigms into a single cooperative system. To overcome inherent conflicts in coordinate spaces, 3D representations, and training objectives, we align both models within a shared canonical space. We employ disentangled cooperative learning, which maintains stable training while enabling seamless collaboration during inference. Specifically, the reconstruction module is adapted to provide canonical geometric anchors, while the diffusion generator leverages latent-augmented conditioning to refine and complete the geometric structure. Experimental results demonstrate that UniRecGen achieves superior fidelity and robustness, outperforming existing methods in creating complete and consistent 3D models from sparse observations. Code is available at https://github.com/zsh523/UniRecGen.

Figures (6)

• Figure 1: Overview of our method. Given $N$ unposed input views, we first canonicalize feed-forward multi-view geometry predictions via branch repurposing and similarity alignment to obtain a canonical point cloud (top). We then train a controllable 3D generator conditioned on this point cloud together with multi-view image features, geometry latents, and camera embeddings to synthesize a high-fidelity mesh (bottom).
• Figure 2: Qualitative Comparison of Canonical Alignment. Our strategy achieves better geometric quality while aligning the canonical object space.
• Figure 3: Qualitative Comparison of Multi-view Condition. Our strategy (right) preserves dense image context more effectively than point-guided sampling (left), leading to improved input alignment.
• Figure 4: Qualitative Comparison on Toys4K and GSO. Compared to state-of-the-art reconstruction and generative baselines, our method produces 3D meshes with higher structural fidelity and superior multi-view consistency from sparse inputs.
• Figure 5: Generalization to Real-world Environments. Our framework demonstrates robustness and superior performance compared to SOTA method.