VTON-HandFit: Virtual Try-on for Arbitrary Hand Pose Guided by Hand Priors Embedding

TL;DR

VTON-HandFit outperforms the baselines in qualitative and quantitative evaluations on the public dataset and the self-collected hand-occlusion Handfit-3K dataset particularly for the arbitrary hand pose occlusion cases in real-world scenarios.

Abstract

Although diffusion-based image virtual try-on has made considerable progress, emerging approaches still struggle to effectively address the issue of hand occlusion (i.e., clothing regions occluded by the hand part), leading to a notable degradation of the try-on performance. To tackle this issue widely existing in real-world scenarios, we propose VTON-HandFit, leveraging the power of hand priors to reconstruct the appearance and structure for hand occlusion cases. Firstly, we tailor a Handpose Aggregation Net using the ControlNet-based structure explicitly and adaptively encoding the global hand and pose priors. Besides, to fully exploit the hand-related structure and appearance information, we propose Hand-feature Disentanglement Embedding module to disentangle the hand priors into the hand structure-parametric and visual-appearance features, and customize a masked cross attention for further decoupled feature embedding. Lastly, we customize a hand-canny constraint loss to better learn the structure edge knowledge from the hand template of model image. VTON-HandFit outperforms the baselines in qualitative and quantitative evaluations on the public dataset and our self-collected hand-occlusion Handfit-3K dataset particularly for the arbitrary hand pose occlusion cases in real-world scenarios. The Code and dataset will be available at \url{https://github.com/VTON-HandFit/VTON-HandFit}.

Figures (7)

• Figure 1: Comparison of different models on virtual try-on with hand occlusion. The leftmost three images are reference images from the VITON-HD test set, showing the target clothing, the model, and a close-up of the model's hand. Our model excels in preserving hand details and achieving realistic clothing transfer, as highlighted in the red boxes. CatVTON and LADI-VTON utilize a parsing model to retain the hand segment, resulting in the inevitable persistence of residual artifacts and backgrounds from the model image.
• Figure 2: An overview of our VTON-HandFit Network. The network consists of two main components: Hand-feature Disentanglement Embedding and Hand-Pose Aggregation Net. The Hand-feature Disentanglement Embedding module uses the HaMer model to extract hand priors, including hand type ${T_h}$, 3D vertices ${V_h}$, spatial joint locations ${J}_{2d}$, and joint rotation matrices $\theta_h$. These features are processed by the Hand-Struct processor to derive structural features $c_{struc}$. Simultaneously, hand images cropped using bounding boxes are processed by DINOv2 and the Hand-Appear processor to obtain visual features $c_{appear}$. The structural and visual features are integrated using mask cross attention. The Hand-Pose Aggregation Net module controls body and hand poses by aggregating DWpose, Densepose, and hand depth maps.
• Figure 3: Qualitative comparisons of VTON-HandFit with other methods on public datasets: (I) Dresscode and (II) VITON-HD.
• Figure 4: Qualitative comparisons of VTON-HandFit with other methods on our Handfit-3K dataset. In Handfit-3K, we remark that the hands in the images are occluded, making it difficult to distinguish them directly through masks.
• Figure 5: Qualitative comparisons in real-world scenarios.