Pragmatist: Multiview Conditional Diffusion Models for High-Fidelity 3D Reconstruction from Unposed Sparse Views

TL;DR

Pragmatist addresses 3D reconstruction from sparse, unposed views by reframing the problem as conditional novel view synthesis. It introduces a multiview conditional diffusion model that generates complete observations in a canonical object coordinate system, enabling robust reconstruction with a feed-forward NeRF/triplane mesh pipeline. A pose-inversion and texture refinement step leverages input views to tighten geometry and appearance, achieving state-of-the-art performance on benchmarks such as GSO and OmniObject3D. By combining generative priors with geometric constraints, Pragmatist effectively mitigates ill-posedness and generalizes well to in-the-wild data.

Abstract

Inferring 3D structures from sparse, unposed observations is challenging due to its unconstrained nature. Recent methods propose to predict implicit representations directly from unposed inputs in a data-driven manner, achieving promising results. However, these methods do not utilize geometric priors and cannot hallucinate the appearance of unseen regions, thus making it challenging to reconstruct fine geometric and textural details. To tackle this challenge, our key idea is to reformulate this ill-posed problem as conditional novel view synthesis, aiming to generate complete observations from limited input views to facilitate reconstruction. With complete observations, the poses of the input views can be easily recovered and further used to optimize the reconstructed object. To this end, we propose a novel pipeline Pragmatist. First, we generate a complete observation of the object via a multiview conditional diffusion model. Then, we use a feed-forward large reconstruction model to obtain the reconstructed mesh. To further improve the reconstruction quality, we recover the poses of input views by inverting the obtained 3D representations and further optimize the texture using detailed input views. Unlike previous approaches, our pipeline improves reconstruction by efficiently leveraging unposed inputs and generative priors, circumventing the direct resolution of highly ill-posed problems. Extensive experiments show that our approach achieves promising performance in several benchmarks.

Figures (8)

• Figure 1: Our method reconstructs 3D geometry and texture from sparse unposed images, even across different scenes of the same object. More results are available in the supplementary material.
• Figure 2: Overview of our pipeline. Our core insight is to generate additional novel views in the canonical object coordinate by conditioning with casual viewpoints to solve sparse unposed view reconstruction. The multi-view conditional generator uses a lightweight encoder to extract input view features and implicitly models the 3D consistency between the input view and the output view in the canonical coordinate system via cross-frame attention. To achieve the trade-off between the input view and the generated prior, the object mesh and triplane features are first obtained using the generated views. The input views are then aligned to the canonical coordinates via diff. rendering, and finally, the surface texture is optimized through the input views.
• Figure 3: Qualitative comparison of unposed sparse-view 3D reconstruction results on the GSO dataset. Our method can produce more detailed and higher-quality reconstruction results than the benchmark method and can support different numbers of unposed inputs. More results can be found in the supplementary material.
• Figure 4: Qualitative comparison of single view 3D reconstruction results on the GSO and in-the-wild datasets. Though our method is not designed for single-view setting, it produces results comparable to these benchmarks and generates reasonable textures in the invisible areas.
• Figure 5: Ablation Studies. The top image illustrates the effectiveness of our reconstruction model, the bottom left image shows the effect of our refinement step, and the bottom right image demonstrates the effect of generative priors.