Deep Feature Deformation Weights

TL;DR

DFD tackles the challenge of achieving semantic, globally coherent deformations at interactive speeds by encoding surface semantics with a neural feature field and deriving deformation weights from feature proximity, eliminating per-handle optimization. It combines a fast barycentric feature distillation pipeline to transfer learned feature fields from coarse renders to high-resolution meshes, with locality, symmetry, and fixed-point constraints that preserve semantic structure. Empirical results show smooth, semantically aware deformations that co-deform semantic parts and preserve symmetry, scaling to meshes with millions of faces on consumer hardware and outperforming baselines in both qualitative and quantitative tests. This approach enables real-time, semantically guided mesh editing, bridging traditional fast, controllable deformation with data-driven semantic priors.

Abstract

Handle-based mesh deformation has been a long-standing paradigm in computer graphics, enabling intuitive shape edits from sparse controls. Classic techniques offer precise and rapid deformation control. However, they solve an optimization problem with constraints defined by control handle placement, requiring a user to know apriori the ideal distribution of handles on the shape to accomplish the desired edit. The mapping from handle set to deformation behavior is often unintuitive and, importantly, non-semantic. Modern data-driven methods, on the other hand, leverage a data prior to obtain semantic edits, but are slow and imprecise. We propose a technique that fuses the semantic prior of data with the precise control and speed of traditional frameworks. Our approach is surprisingly simple yet effective: deep feature proximity makes for smooth and semantic deformation weights, with no need for additional regularization. The weights can be computed in real-time for any surface point, whereas prior methods require optimization for new handles. Moreover, the semantic prior from deep features enables co-deformation of semantic parts. We introduce an improved feature distillation pipeline, barycentric feature distillation, which efficiently uses the visual signal from shape renders to minimize distillation cost. This allows our weights to be computed for high resolution meshes in under a minute, in contrast to potentially hours for both classical and neural methods. We preserve and extend properties of classical methods through feature space constraints and locality weighting. Our field representation allows for automatic detection of semantic symmetries, which we use to produce symmetry-preserving deformations. We show a proof-of-concept application which can produce deformations for meshes up to 1 million faces in real-time on a consumer-grade machine.

Figures (21)

• Figure 1: Our DFD framework enables flexible control over a wide range of deformations in real time. Symmetric deformations may be achieved through our automatically detected symmetry plane (yellow).
• Figure 2: General Affine Transformations. DFD weights effectively interpolate affine transformations to generate plausible pose changes. We can generate a variety of deformations by leveraging detected symmetries (a,b) (\ref{['subsec:symmetry']}) and localization control (b, d) (\ref{['subsec:locality']}).
• Figure 3: Translation Edits. Edit results from different handle (blue) translations to target locations (green) using translations prescribed by APAP-Bench 3D. yoo2024plausible. Weights are visualized as heatmap insets. A larger set of results from the dataset are shown in the supplemental.
• Figure 4: Barycentric Feature Distillation. (Left) Existing feature distillation methods use only pixels intersected by raster vertices. Barycentric distillation takes advantage of the known geometry to supervise with features at all pixels intersected by a triangle. (Right) High resolution meshes look visually unchanged even with extreme reduction using QEM (99%). Consequently their feature fields are virtually identical (PCA insets). We opt to distill features using renders of low-resolution meshes, and use them to deform meshes at their original resolution.
• Figure 5: Semantic Symmetry Detection. Our neural field representation returns semantic features for arbitrary 3D points. This enables us to evaluate candidate symmetry planes on points away from the shape surface. This enables identification of symmetry planes where shape features are semantically reflected but not necessarily geometrically. Our symmetric deformations are generated by manipulating only one side of the shape.