DragD3D: Realistic Mesh Editing with Rigidity Control Driven by 2D Diffusion Priors
Tianhao Xie, Eugene Belilovsky, Sudhir Mudur, Tiberiu Popa
TL;DR
DragD3D addresses realistic mesh editing by enabling vertex-level control with global context through 2D diffusion priors. It introduces a rotation–stretch decomposition of the neural Jacobian field and fuses a 3D geometric regularizer with Delta Denoising Score guidance from a diffusion model, enabling class-agnostic, context-aware deformations from few handles. It adds user controls like a rigidity mask and per-triangle weights, and uses random viewpoints with a differentiable renderer to optimize the deformation. Experiments show improved realism and feature preservation over purely optimization-based methods, with ablations highlighting the value of DDS and geometric regularization. The approach demonstrates a practical pathway to integrate large-scale 2D priors into 3D mesh editing without requiring 3D training data.
Abstract
Direct mesh editing and deformation are key components in the geometric modeling and animation pipeline. Mesh editing methods are typically framed as optimization problems combining user-specified vertex constraints with a regularizer that determines the position of the rest of the vertices. The choice of the regularizer is key to the realism and authenticity of the final result. Physics and geometry-based regularizers are not aware of the global context and semantics of the object, and the more recent deep learning priors are limited to a specific class of 3D object deformations. Our main contribution is a vertex-based mesh editing method called DragD3D based on (1) a novel optimization formulation that decouples the rotation and stretch components of the deformation and combines a 3D geometric regularizer with (2) the recently introduced DDS loss which scores the faithfulness of the rendered 2D image to one from a diffusion model. Thus, our deformation method achieves globally realistic shape deformation which is not restricted to any class of objects. Our new formulation optimizes directly the transformation of the neural Jacobian field explicitly separating the rotational and stretching components. The objective function of the optimization combines the approximate gradients of DDS and the gradients from the geometric loss to satisfy the vertex constraints. Additional user control over desired global shape deformation is made possible by allowing explicit per-triangle deformation control as well as explicit separation of rotational and stretching components of the deformation. We show that our deformations can be controlled to yield realistic shape deformations that are aware of the global context of the objects, and provide better results than just using geometric regularizers.