Garment Animation NeRF with Color Editing

TL;DR

This paper proposes a novel approach to directly synthesize garment animations from body motion sequences without the need for an explicit garment proxy, and demonstrates the generalizability of the method across unseen body motions and camera views, ensuring detailed structural consistency.

Abstract

Generating high-fidelity garment animations through traditional workflows, from modeling to rendering, is both tedious and expensive. These workflows often require repetitive steps in response to updates in character motion, rendering viewpoint changes, or appearance edits. Although recent neural rendering offers an efficient solution for computationally intensive processes, it struggles with rendering complex garment animations containing fine wrinkle details and realistic garment-and-body occlusions, while maintaining structural consistency across frames and dense view rendering. In this paper, we propose a novel approach to directly synthesize garment animations from body motion sequences without the need for an explicit garment proxy. Our approach infers garment dynamic features from body motion, providing a preliminary overview of garment structure. Simultaneously, we capture detailed features from synthesized reference images of the garment's front and back, generated by a pre-trained image model. These features are then used to construct a neural radiance field that renders the garment animation video. Additionally, our technique enables garment recoloring by decomposing its visual elements. We demonstrate the generalizability of our method across unseen body motions and camera views, ensuring detailed structural consistency. Furthermore, we showcase its applicability to color editing on both real and synthetic garment data. Compared to existing neural rendering techniques, our method exhibits qualitative and quantitative improvements in garment dynamics and wrinkle detail modeling. Code is available at \url{https://github.com/wrk226/GarmentAnimationNeRF}.

Figures (10)

• Figure 1: The architecture of Garment Animation NeRF. Given a sequence of character's body motion, we construct a neural radiance field to render animation of the character dressed in the target garment. We first employ a dynamic feature encoder $\mathcal{E}$ to infer garment dynamic feature map $F_t^s$ from the information textures $V_t$ and $N_t$ of the body motion $B_t$. Simultaneously, taking body neural texture images $Q_t$ of the body at front $c^{fr}$ and back views $c^{ba}$, we use a pre-trained image generator $\mathcal{G}_{img}$ to predict reference images $\hat{I}_t(c^{fr})$ and $\hat{I}_t(c^{ba})$. Subsequently, we use the detail feature encoder $\mathcal{G}_{en}$ to generate the detail feature maps $F_t^d(c^{fr})$ and $F_t^d(c^{ba})$. Then, we obtain body-aware geometric information $x_t^b$ by calculating the distance $h$ between sampling points and the body surface, and finding $b_o$ on the canonical body shape. Finally, we utilize a NeRF network $\mathcal{M}_{NeRF}$ to render garment appearance feature image $F_t^{\zeta}$. To enable color editing, we introduce a network $\mathcal{M}_D$ to decompose the garment appearance into a front mask $M_t$, a color offset map $O_t$, a radiance map $R_t$ and a blending weight map $W_t$. By linearly recombining those visual elements, we synthesize the final frame image $I_t$. Except for the generator $\mathcal{G}_{img}$, we jointly train the networks of $\mathcal{E}$, $\mathcal{G}_{en}$, $\mathcal{M}_{NeRF}$ and $\mathcal{M}_{D}$ in an end-to-end manner.
• Figure 2: Unseen view. Our model can generate a structural consistent garment appearance from arbitrary viewpoint.
• Figure 3: Unseen motion. We test our model on various garments using several motion sequences that were not seen in the training process.
• Figure 4: Unseen body shapes. Without fine-tuning, our method can synthesize target garment animation that fit with the body shapes (round and thin) different from the original training data body shape.
• Figure 5: Real-captured data. Our model can be trained with real-captured data and generalize to the body motion unseen during the training.