Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

As-Plausible-As-Possible: Plausibility-Aware Mesh Deformation Using 2D Diffusion Priors

Seungwoo Yoo, Kunho Kim, Vladimir G. Kim, Minhyuk Sung

TL;DR

APAP solves plausibility-aware mesh deformation by coupling a differentiable Poisson-based Jacobian representation with a finetuned 2D diffusion prior via Score Distillation Sampling. The method operates in two stages: first updating a Jacobian field to satisfy user handle constraints, then refining with a diffusion-prior-guided loss to preserve perceptual plausibility while honoring edits. By finetuning the diffusion model with LoRA and leveraging multiple viewpoints, APAP achieves superior 2D/3D deformation results compared to purely geometric baselines like ARAP, as demonstrated on APAP-Bench with both qualitative and quantitative gains. The approach provides a scalable, generalizable framework for interactive shape editing that integrates classical geometry processing with modern diffusion priors, enabling realistic deformations across diverse object categories.

Abstract

We present As-Plausible-as-Possible (APAP) mesh deformation technique that leverages 2D diffusion priors to preserve the plausibility of a mesh under user-controlled deformation. Our framework uses per-face Jacobians to represent mesh deformations, where mesh vertex coordinates are computed via a differentiable Poisson Solve. The deformed mesh is rendered, and the resulting 2D image is used in the Score Distillation Sampling (SDS) process, which enables extracting meaningful plausibility priors from a pretrained 2D diffusion model. To better preserve the identity of the edited mesh, we fine-tune our 2D diffusion model with LoRA. Gradients extracted by SDS and a user-prescribed handle displacement are then backpropagated to the per-face Jacobians, and we use iterative gradient descent to compute the final deformation that balances between the user edit and the output plausibility. We evaluate our method with 2D and 3D meshes and demonstrate qualitative and quantitative improvements when using plausibility priors over geometry-preservation or distortion-minimization priors used by previous techniques. Our project page is at: https://as-plausible-aspossible.github.io/

As-Plausible-As-Possible: Plausibility-Aware Mesh Deformation Using 2D Diffusion Priors

TL;DR

APAP solves plausibility-aware mesh deformation by coupling a differentiable Poisson-based Jacobian representation with a finetuned 2D diffusion prior via Score Distillation Sampling. The method operates in two stages: first updating a Jacobian field to satisfy user handle constraints, then refining with a diffusion-prior-guided loss to preserve perceptual plausibility while honoring edits. By finetuning the diffusion model with LoRA and leveraging multiple viewpoints, APAP achieves superior 2D/3D deformation results compared to purely geometric baselines like ARAP, as demonstrated on APAP-Bench with both qualitative and quantitative gains. The approach provides a scalable, generalizable framework for interactive shape editing that integrates classical geometry processing with modern diffusion priors, enabling realistic deformations across diverse object categories.

Abstract

We present As-Plausible-as-Possible (APAP) mesh deformation technique that leverages 2D diffusion priors to preserve the plausibility of a mesh under user-controlled deformation. Our framework uses per-face Jacobians to represent mesh deformations, where mesh vertex coordinates are computed via a differentiable Poisson Solve. The deformed mesh is rendered, and the resulting 2D image is used in the Score Distillation Sampling (SDS) process, which enables extracting meaningful plausibility priors from a pretrained 2D diffusion model. To better preserve the identity of the edited mesh, we fine-tune our 2D diffusion model with LoRA. Gradients extracted by SDS and a user-prescribed handle displacement are then backpropagated to the per-face Jacobians, and we use iterative gradient descent to compute the final deformation that balances between the user edit and the output plausibility. We evaluate our method with 2D and 3D meshes and demonstrate qualitative and quantitative improvements when using plausibility priors over geometry-preservation or distortion-minimization priors used by previous techniques. Our project page is at: https://as-plausible-aspossible.github.io/
Paper Structure (32 sections, 7 equations, 8 figures, 4 tables, 1 algorithm)

This paper contains 32 sections, 7 equations, 8 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Geometric Mesh Deformation
  4. Data-Driven Mesh Deformation
  5. Pretrained 2D Priors for Shape Manipulation
  6. Method
  7. Representing Shapes as Jacobian Fields
  8. Score Distillation for Shape Deformation
  9. As-Plausible-As-Possible (APAP)
  10. Experiments
  11. Experiment Setup
  12. Benchmark.
  13. Baselines.
  14. Evaluation Metrics.
  15. 3D Shape Deformation
  16. ...and 17 more sections

Figures (8)

  • Figure 1: The overview of APAP.APAP parameterizes a triangular mesh as a per-face Jacobian field that can be updated via gradient-descent. Given a textured mesh and user inputs specifying the handle(s) and anchor(s), our framework initializes a Jacobian field as a trainable parameter. During the first stage, the Jacobian field is updated via iterative optimization of $\mathcal{L}_h$, a soft constraint that initially deforms the shape according to the user's instruction. In the following stage, the mesh is rendered using a differentiable renderer $\mathcal{R}$ and the rendered image is provided as an input to a diffusion prior finetuned with LoRA Hu:2022LoRA that computes the SDS loss $\mathcal{L}_{\text{SDS}}$. The joint optimization of $\mathcal{L}_{h}$ and $\mathcal{L}_{\text{SDS}}$ improves the visual plausibility of the mesh while conforming to the given edit instruction.
  • Figure 2: Qualitative results from 3D shape deformation. We visualize the source shapes and their deformations made using ARAP Sorkine:2007ARAP and ours by following the instructions each of which specifies a handle (red), an edit direction denoted with an arrow (gray), and an anchor (green). We showcase the rendered images captured from two different viewpoints, as well as one zoom-in view highlighting local details.
  • Figure 3: Failure cases of DragDiffusion. DragDiffusion Shi:2023DragDiffusion can easily compromise the identity of edited instances as it manipulates their latents without an explicit parameterization, the identity of instances can be broken during editing.
  • Figure 4: Qualitative results from 2D mesh deformation. 2D meshes are edited using ARAP Sorkine:2007ARAP and the proposed method following the edit instruction consisting of a handle (red), a target direction (gray), and an anchor (green). We showcase the rendered images of the edited meshes, as well as a zoom-in view highlighting local details.
  • Figure A5: Examples of questionnaires displayed during the user study (2D mesh editing). During the user study, we asked the participants to evaluate 20 different result pairs from ARAP Sorkine:2007ARAP and ours as shown on the left. To check whether a participant is focusing on the user study, we included 5 items for the vigilance test. As shown on the right, a question for the vigilance test includes an image of an object that is not related to the source image.
  • ...and 3 more figures