Patch-based Representation and Learning for Efficient Deformation Modeling

TL;DR

PolyFit introduces a patch-based, jet-fitting representation for surfaces that yields a compact, differentiable deformation state. By combining ACVD patching, PCA-based canonicalization, and a rigid STN, it enables efficient deformation via a small set of jet coefficients and supports two core applications: PolySfT, a fast test-time optimization for shape-from-template, and OneFit, a self-supervised, mesh- and garment-agnostic draping model. Across extensive experiments, PolyFit achieves accurate surface fitting with lower Chamfer distances than AtlasNet, while PolySfT and OneFit deliver competitive reconstruction quality and substantially faster inference compared with physics-based or mesh-specific baselines. The work demonstrates the practicality of polynomial, patch-wise representations for scalable, cross-resolution deformation modeling in vision and graphics, with limitations including single-valued patch height and potential seam artifacts that are ripe for future improvements.

Abstract

In this paper, we present a patch-based representation of surfaces, PolyFit, which is obtained by fitting jet functions locally on surface patches. Such a representation can be learned efficiently in a supervised fashion from both analytic functions and real data. Once learned, it can be generalized to various types of surfaces. Using PolyFit, the surfaces can be efficiently deformed by updating a compact set of jet coefficients rather than optimizing per-vertex degrees of freedom for many downstream tasks in computer vision and graphics. We demonstrate the capabilities of our proposed methodologies with two applications: 1) Shape-from-template (SfT): where the goal is to deform the input 3D template of an object as seen in image/video. Using PolyFit, we adopt test-time optimization that delivers competitive accuracy while being markedly faster than offline physics-based solvers, and outperforms recent physics-guided neural simulators in accuracy at modest additional runtime. 2) Garment draping. We train a self-supervised, mesh- and garment-agnostic model that generalizes across resolutions and garment types, delivering up to an order-of-magnitude faster inference than strong baselines.

Figures (15)

• Figure 1: Patch-based representation and learning. Against conventional per-vertex parameterizations of mesh deformation (a), we present PolyFit (b) which obtains a simplified patch-wise representation by fitting jet functions with limited parameters; consequently simplifying the transformations to modification of patch parameters only (c) and reducing the computational overhead by a large margin.
• Figure 2: a) PolyFit. It orients the input patch to improve bijectivity with the uv-plane and obtains an analytic representation using n-jet fitting. b) PolySfT. Using PolyFit, the input 3D template is deformed to match input images by estimating the offsets of fitted jet coefficients $\Delta \alpha$, uv displacements $\Delta uv$, as well as a rigid transformation $(\mathbf{R}_c,\mathbf{T}_c)$.
• Figure 3: OneFit. It deforms $\mathcal{T}$ isometrically to obtain $\mathcal{P}$ posed on body $\mathcal{B}_t$ by forcing patch boundary consistency, avoiding collisions and maintaining physical equilibria.
• Figure 4: Error map comparison between DeepSfT and Ours on example frames from the Kinect-Paper dataset.
• Figure 5: Single garment OneFit under garment intra-class variations. Trained on a Tank top (in green), OneFit is able to drape tank tops of different styles.