Deformable 3D Shape Diffusion Model

TL;DR

This work introduces the Deformable 3D Shape Diffusion Model, a geometry-aware diffusion framework that replaces standard Gaussian diffusion with a Differential Deformation Kernel to progressively deform a source shape toward a target template. By coupling forward diffusion with a Deformable Imitation Learning objective, the model enables robust non-rigid deformation for point clouds, meshes, and facial animation, achieving state-of-the-art results for point cloud generation and competitive performance for mesh deformation. The approach relies on a suite of shape regularizations (Chamfer, Normal Consistency, Laplacian, Edge Length, and Potential Energy) to preserve topology and spatial coherence during diffusion, and introduces optimized initialization and equispaced sampling to accelerate convergence. Empirical results across high-resolution point clouds and well-manifold meshes demonstrate strong quality with fewer diffusion steps, and qualitative demonstrations highlight potential applications in rendering and expressive animation, suggesting broad utility in graphics, VR, and interactive design.

Abstract

The Gaussian diffusion model, initially designed for image generation, has recently been adapted for 3D point cloud generation. However, these adaptations have not fully considered the intrinsic geometric characteristics of 3D shapes, thereby constraining the diffusion model's potential for 3D shape manipulation. To address this limitation, we introduce a novel deformable 3D shape diffusion model that facilitates comprehensive 3D shape manipulation, including point cloud generation, mesh deformation, and facial animation. Our approach innovatively incorporates a differential deformation kernel, which deconstructs the generation of geometric structures into successive non-rigid deformation stages. By leveraging a probabilistic diffusion model to simulate this step-by-step process, our method provides a versatile and efficient solution for a wide range of applications, spanning from graphics rendering to facial expression animation. Empirical evidence highlights the effectiveness of our approach, demonstrating state-of-the-art performance in point cloud generation and competitive results in mesh deformation. Additionally, extensive visual demonstrations reveal the significant potential of our approach for practical applications. Our method presents a unique pathway for advancing 3D shape manipulation and unlocking new opportunities in the realm of virtual reality.

Figures (11)

• Figure 1: Illustration of the geometric imitation procedure (4th row). First, we diffuse a given source shape $\mathcal{X}^{(0)}$ to a well-initialized template shape $\mathcal{X}^{(T)}$ using DDK in a sequence $\{\mathcal{X}^{(0)}, \mathcal{X}^{(1)}, \cdots \mathcal{X}^{(T)} \}$. Subsequently, we use the DDM to reverse the diffusion process and obtain the desired shape. Our proposed method is versatile and can be applied to various tasks such as point cloud generation (1st row), mesh deformation (2nd row), and facial animation (3rd row).
• Figure 2: Comparison of diffusion procedures guided by GDK and DDK. As depicted in the first row, using only GDK introduces Gaussian noise to vertices at each step, which quickly erodes the geometric structure of the shape, resulting in disordered point clouds where the edges intersect, thereby disrupting the topology of the meshes. In contrast, the proposed DDK can deform shapes based on multiple shape regularizations, producing well-structured manifold meshes.
• Figure 3: Comparison of different point cloud generation methods, namely, Atlas Net groueix2018papier, Shape GF cai2020learning, and DPM3D luo2021diffusion.
• Figure 4: Comparative analysis of various mesh generation methodologies, namely, Pixel2Mesh wang2018pixel2mesh, R2N2 choy20163d, PCGN fan2017point, OCC Net mescheder2019occupancy, Mesh RCNN gkioxari2019mesh, Shape GF cai2020learning, Atlas Net groueix2018papier, and NMF gupta2020neural.
• Figure 5: High-fidelity image rendered with the generated pillow mesh.