View-Dependent Deformation Fields for 2D Editing of 3D Models

TL;DR

This paper addresses editing 3D models to match 2D edits from specific viewpoints by introducing view-dependent deformation fields that map 3D points through a view-conditioned deformation. The core idea lifts per-view 2D deformations to 3D space and blends them in a layered, compositional manner, enabling complex, view-continuous edits on both 3D Gaussian Splats and meshes. Key contributions include (1) a general f_v construction from 2D keyframes, (2) a non-linear, sequential composition scheme for multiple deformations, (3) a practical method to apply 3D deformations to 3D Gaussian Splats via a first-order affine approximation, and (4) a 2D-deformation authoring pipeline using 2D meshes and BBW rigging. The approach achieves expressive, view-dependent stylization, occlusion handling, and fitting to 2D illustrations, with scalable efficiency suitable for high-resolution models and potential extensions to animation and symmetric deformations.

Abstract

We propose a method for authoring non-realistic 3D objects (represented as either 3D Gaussian Splats or meshes), that comply with 2D edits from specific viewpoints. Namely, given a 3D object, a user chooses different viewpoints and interactively deforms the object in the 2D image plane of each view. The method then produces a "deformation field" - an interpolation between those 2D deformations in a smooth manner as the viewpoint changes. Our core observation is that the 2D deformations do not need to be tied to an underlying object, nor share the same deformation space. We use this observation to devise a method for authoring view-dependent deformations, holding several technical contributions: first, a novel way to compositionality-blend between the 2D deformations after lifting them to 3D - this enables the user to "stack" the deformations similarly to layers in an editing software, each deformation operating on the results of the previous; second, a novel method to apply the 3D deformation to 3D Gaussian Splats; third, an approach to author the 2D deformations, by deforming a 2D mesh encapsulating a rendered image of the object. We show the versatility and efficacy of our method by adding cartoonish effects to objects, providing means to modify human characters, fitting 3D models to given 2D sketches and caricatures, resolving occlusions, and recreating classic non-realistic paintings as 3D models.

Figures (19)

• Figure 1: Creating a single "2D" deformation of a 3D object. A 3D model (left) is rendered from a chosen view point $v_i$ into a 2D image (middle), which is meshed. The user selects deformation handles (magenta circles) and drags them ("user edit"), creating a deformation of the 2D mesh (right), which defines a 2D deformation $\phi_i$ and is lifted to a 3D deformation $\Phi_i$ (dashed square).
• Figure 2: Creating a view-dependent deformation field from several 2D deformations. A 2D mesh of the object is created for each of the keypoint views $v_1,v_2,v_3$. The meshes are deformed by the user to define three 2D deformations, $\phi_1,\phi_2,\phi_3$. When viewed from another view, $v^*$, the three deformations are lifted to 3D and interpolated, using the compositional interpolation formula, Equation \ref{['eq:interpolation_point']}, to yield the 3D deformation $f_{v^*}$.
• Figure 3: view-dependent positioning of facial features. Our method enables artists to design characters whose facial features change position and shape as the camera changes its viewpoint. In this example we emulate an edit similar to the one shown by the custom system designed for the Peanuts movie peanuts.
• Figure 4: Reproducing Van Gogh's chair as a view-dependent 3D model. Our method enables a user to fit a 3D chair to Van Gogh's painting, breaking perspective rules while still appearing correct from other views.
• Figure 5: Still life in the style of Cézanne. We emulate Cézanne's still life paintings, famous as one of the first examples of objects presented in multiple perspectives at the same time.