Generative Shape Reconstruction with Geometry-Guided Langevin Dynamics

Abstract

Reconstructing complete 3D shapes from incomplete or noisy observations is a fundamentally ill-posed problem that requires balancing measurement consistency with shape plausibility. Existing methods for shape reconstruction can achieve strong geometric fidelity in ideal conditions but fail under realistic conditions with incomplete measurements or noise. At the same time, recent generative models for 3D shapes can synthesize highly realistic and detailed shapes but fail to be consistent with observed measurements. In this work, we introduce GG-Langevin: Geometry-Guided Langevin dynamics, a probabilistic approach that unifies these complementary perspectives. By traversing the trajectories of Langevin dynamics induced by a diffusion model, while preserving measurement consistency at every step, we generatively reconstruct shapes that fit both the measurements and the data-informed prior. We demonstrate through extensive experiments that GG-Langevin achieves higher geometric accuracy and greater robustness to missing data than existing methods for surface reconstruction.

Figures (14)

• Figure 1: GG-Langevin. We combine the prior learned by a diffusion model with gradients from a geometric loss at inference time. By guiding the trajectories of Langevin dynamics, we obtain shapes that are both measurement-consistent and prior-consistent.
• Figure 2: Blue trajectories: Non-guided Langevin dynamics on the prior distribution $p(z)$, initialized at an incomplete shape using the VAE encoder $z_0 = E(\mathcal{P})$. It generates plausible, complete shapes but quickly drifts from the measurements. Green trajectory: GG-Langevin generatively reconstructs the shape from the input point cloud. By incorporating gradients from a geometric loss $\mathcal{L}(z,\mathcal{P})$, it keeps the sampling trajectory close to the manifold of measurement-consistent shapes where $\mathcal{L}(z,\mathcal{P}) = 0$ (indicated by the dashed red line). On the right: A side-by-side comparison of sampling trajectories from Langevin dynamics and Geometry-Guided Langevin dynamics.
• Figure 3: Toy example. Demonstration that our method generates samples from the geometry-guided distribution $\tilde{p}(z|\mathcal{P})$. Blue: Data distribution. Red: Two variants of guidance weight. Green: Geometry-guided product distribution. Solid lines indicate the predicted closed-form distributions. The histograms show samples generated using regular Langevin dynamics and GG-Langevin, respectively. We train an MLP diffusion model on samples from a bimodal Gaussian $p(z)$. The samples closely follow the predicted distributions.
• Figure 4: Reconstruction results on sparse point clouds. Provided sparse point cloud scans as input, GG-Langevin recovers the complete surface and fine structures, significantly improving the reconstruction accuracy in comparison to previous work.
• Figure 5: Reconstruction results on incomplete point clouds. Despite incomplete point cloud scans as input, GG-Langevin recovers the missing structure with prior-consistent geometry. By comparison, previous work either struggles to complete the geometry or hallucinates implausible completions.