Pattern Guided UV Recovery for Realistic Video Garment Texturing

TL;DR

This paper presents Pattern Guided UV Recovery for Realistic Video Garment Texturing, a pipeline that enables automatic texture replacement on garments from single-view videos. It combines pattern-guided correspondence detection with per-pixel UV regression using a blended-weight MLP, augmented by a Jacobian-based UV loss and a gradient-directed regularization to preserve folds and seams. Temporal consistency is enforced through a blended temporal model and inpainting/flow-based constraints, while shading and UV-gradient prediction enable photorealistic texture synthesis. The approach demonstrates robust performance across diverse garments, lighting, and motions, with quantitative gains over baselines and potential to scale fashion e-commerce workflows for one-click texture customization.

Abstract

The fast growth of E-Commerce creates a global market worth USD 821 billion for online fashion shopping. What unique about fashion presentation is that, the same design can usually be offered with different cloths textures. However, only real video capturing or manual per-frame editing can be used for virtual showcase on the same design with different textures, both of which are heavily labor intensive. In this paper, we present a pattern-based approach for UV and shading recovery from a captured real video so that the garment's texture can be replaced automatically. The core of our approach is a per-pixel UV regression module via blended-weight multilayer perceptrons (MLPs) driven by the detected discrete correspondences from the cloth pattern. We propose a novel loss on the Jacobian of the UV mapping to create pleasant seams around the folding areas and the boundary of occluded regions while avoiding UV distortion. We also adopts the temporal constraint to ensure consistency and accuracy in UV prediction across adjacent frames. We show that our approach is robust to a variety type of clothes, in the wild illuminations and with challenging motions. We show plausible texture replacement results in our experiment, in which the folding and overlapping of the garment can be greatly preserved. We also show clear qualitative and quantitative improvement compared to the baselines as well. With the one-click setup, we look forward to our approach contributing to the growth of fashion E-commerce.

Figures (20)

• Figure 1: We present an approach for pattern guided garment texture replacement in product virtual showcase. Given a video of a person wearing the target garments fabricated with our designed patterns, our method automatically extracts per-pixel UV coordinates for the garments in each frame as well as the shading and mask layers so that different texture maps can be applied to the image space for fast realistic visualization with just one click. The quality of our results lives up to the requirements from commercial applications where challenging areas such as folding areas and seams are nicely handled.
• Figure 2: What unique about fashion E-commerce is that, a product is commonly designed once but fabricated with a variety of optional patterns, i.e., the same shirt can be made of cornflower blue or maroon red. Image source: Dior© dior, Charles Tyrwhitt© charlestyrwhitt.
• Figure 3: Pipeline overview. Starting from an input frame $I_i$, we perform correspondence detection (Section \ref{['sec:corr']}) to establish discrete correspondences between the image frame and the template pattern. Input coordinates are fed into the blended weight MLP (Section \ref{['sec:temp']}) to regress the per-pixel UV coordinates, with both data constraint and gradient constraint (Section \ref{['sec:dense']}). We also apply temporal constraints to neighboring frames to ensure the visual consistency (Section \ref{['sec:temp']}). The shading and mask are also predicted from the input (Section \ref{['sec:shading']}). The predicted UVs are used to texture the albedo layer with given texture maps, and the final images can be composed accordingly.
• Figure 4: Our correspondence detection enhances the results from GarmentAvatar halimi2022garment (yellow dots) and produces $19\%$ more correspondences (green dots). We present the boundary case (top row) and the motion blur case (bottom row) to show the efficacy of the improved detection algorithm.
• Figure 5: The green dots represent the correspondences, and the regions not covered by yellow color are $\Theta$, which means correspondences are missing here. Correspondences could be difficult to be detected at $\Theta$ due to wrinkles and occlusions. We apply the $L_{0.5}$ gradient loss on $\Theta$ to constrain UV. Such constraint avoids UV smooth transition and produces discontinuity of UV at the folding and occlusion areas.