Geometry in Style: 3D Stylization via Surface Normal Deformation

TL;DR

Geometry in Style is presented, a new method for identity-preserving mesh stylization that recovers deformations from target normals that are expressive enough to enable detailed stylizations yet restrictive enough to preserve the shape’s identity.

Abstract

We present Geometry in Style, a new method for identity-preserving mesh stylization. Existing techniques either adhere to the original shape through overly restrictive deformations such as bump maps or significantly modify the input shape using expressive deformations that may introduce artifacts or alter the identity of the source shape. In contrast, we represent a deformation of a triangle mesh as a target normal vector for each vertex neighborhood. The deformations we recover from target normals are expressive enough to enable detailed stylizations yet restrictive enough to preserve the shape's identity. We achieve such deformations using our novel differentiable As-Rigid-As-Possible (dARAP) layer, a neural-network-ready adaptation of the classical ARAP algorithm which we use to solve for per-vertex rotations and deformed vertices. As a differentiable layer, dARAP is paired with a visual loss from a text-to-image model to drive deformations toward style prompts, altogether giving us Geometry in Style. Our project page is at https://threedle.github.io/geometry-in-style.

Figures (18)

• Figure 1: Our method deforms a source shape (gray) into a text-specified semantic style (blue). While the deformations are expressive, they preserve the identity of the original shape.
• Figure 2: Style diversity. Our method is capable of deforming various input meshes towards a variety of text-specified styles. The style can be manifested as fine geometric details, like in the ornate art deco column, or as low-frequency deformations, such as the joints of the cybernetic glove. Our method retains the structural features of the input shape, such as a flat arm on the antique sofa. Moreover, the resultant stylizations are in accordance with prompt semantics and part-aware semantics: the folds in the tropical chair are on the seat and backrest as opposed to the legs, the head of the penguin becomes like the top of a fire hydrant, and the racer bunny's thigh turns into the shape of a wheel.
• Figure 3: Overview of our stylization pipeline. Geometry in Style optimizes vertex normals to deform the mesh surface, subject to a stylization text prompt. Using the normals undergoing optimization as a target for our differentiable As-Rigid-As-Possible method (dARAP), the dARAP local step computes a rotation matrix per vertex; we then obtain the deformed surface via our dARAP global solve. Then, we utilize a differentiable renderer and a diffusion model-based semantic loss to guide the normals being optimized towards a deformation matching the desired style prompt.
• Figure 4: Local Orthogonal Procrustes. The single-iteration local step of our dARAP energy solves for the best fit rotation given the original and and target normal.
• Figure 5: Our method is capable of deforming the same mesh towards different text-specified styles.