ReconViaGen: Towards Accurate Multi-view 3D Object Reconstruction via Generation

TL;DR

ReconViaGen tackles the problem of incomplete multi-view 3D object reconstruction under occlusions and sparse coverage by integrating a strong 2D/3D reconstruction prior (VGGT) with diffusion-based 3D generative priors (TRELLIS). It introduces a coarse-to-fine pipeline conditioned by global geometry and per-view tokens, augmented with a rendering-aware velocity compensation to enforce pixel-level alignment, and it optimizes the generation using a conditional flow matching objective $\mathcal{L}_{CFM}(\theta)$ plus reconstruction losses. The key contributions are three mechanisms: global geometry conditioning (GGC), per-view conditioning (PVC), and rendering-aware velocity compensation (RVC), which together yield high fidelity in both global structure and local texture details. Experiments on Dora-Bench and OmniObject3D demonstrate state-of-the-art performance and robustness to varied input views, enabling accurate, complete reconstructions even with occlusions.

Abstract

Existing multi-view 3D object reconstruction methods heavily rely on sufficient overlap between input views, where occlusions and sparse coverage in practice frequently yield severe reconstruction incompleteness. Recent advancements in diffusion-based 3D generative techniques offer the potential to address these limitations by leveraging learned generative priors to hallucinate invisible parts of objects, thereby generating plausible 3D structures. However, the stochastic nature of the inference process limits the accuracy and reliability of generation results, preventing existing reconstruction frameworks from integrating such 3D generative priors. In this work, we comprehensively analyze the reasons why diffusion-based 3D generative methods fail to achieve high consistency, including (a) the insufficiency in constructing and leveraging cross-view connections when extracting multi-view image features as conditions, and (b) the poor controllability of iterative denoising during local detail generation, which easily leads to plausible but inconsistent fine geometric and texture details with inputs. Accordingly, we propose ReconViaGen to innovatively integrate reconstruction priors into the generative framework and devise several strategies that effectively address these issues. Extensive experiments demonstrate that our ReconViaGen can reconstruct complete and accurate 3D models consistent with input views in both global structure and local details.Project page: https://jiahao620.github.io/reconviagen.

Figures (7)

• Figure 1: In the task of 3D object reconstruction from multi-view images, existing pure reconstruction methods can only produce incomplete results, while generation-based methods can get plausible complete results but with strong inconsistency with input images. Our ReconViaGen integrates 3D reconstruction and diffusion-based generation priors into one framework that leads to accurate reconstructions.
• Figure 2: An overview illustration of the proposed ReconViaGen framework, which integrates strong reconstruction priors with 3D diffusion-based generation priors for accurate reconstruction at both the global and local level.
• Figure 3: Reconstruction result comparisons between our ReconViaGen and other baseline methods on samples from the Dora-bench and OminiObject3D datasets. Zoom in for better visualization.
• Figure 4: Reconstruction results on in-the-wild samples. Note that commercial 3D generators require input images from orthogonal viewpoints, while ours can accept views from arbitrary camera poses for robust outputs. Zoom in for better visualization in detail.
• Figure 5: Qualitative comparisons for different variants of ReconViaGen for ablative study. Zoom in for better visualization in detail.