A Novel Garment Transfer Method Supervised by Distilled Knowledge of Virtual Try-on Model

TL;DR

This work tackles garment transfer by leveraging knowledge distillation from virtual try-on to supervise transfer parsing and garment warping. A teacher-student framework transfers shape-aware parsing and high-fidelity warping, with a self-study loop to correct teaching errors, and an arm regrowth module to preserve body features. The method introduces a transfer parsing prior and a progressive flow with STN-based initial alignment and shape-content-aware warping to precisely map the garment from one person to another. Empirical results show state-of-the-art performance in preserving garment texture and body features, with strong generalization to new poses and datasets.

Abstract

This paper proposes a novel garment transfer method supervised with knowledge distillation from virtual try-on. Our method first reasons the transfer parsing to provide shape prior to downstream tasks. We employ a multi-phase teaching strategy to supervise the training of the transfer parsing reasoning model, learning the response and feature knowledge from the try-on parsing reasoning model. To correct the teaching error, it transfers the garment back to its owner to absorb the hard knowledge in the self-study phase. Guided by the transfer parsing, we adjust the position of the transferred garment via STN to prevent distortion. Afterward, we estimate a progressive flow to precisely warp the garment with shape and content correspondences. To ensure warping rationality, we supervise the training of the garment warping model using target shape and warping knowledge from virtual try-on. To better preserve body features in the transfer result, we propose a well-designed training strategy for the arm regrowth task to infer new exposure skin. Experiments demonstrate that our method has state-of-the-art performance compared with other virtual try-on and garment transfer methods in garment transfer, especially for preserving garment texture and body features.

Figures (24)

• Figure 1: Garment transfer vs virtual try-on.
• Figure 2: The outline of the proposed method. We distill the parsing reasoning and garment warping knowledge in parsing reasoning and garment warping. Following training, ${{{\cal S}_p}}$ is capable of reasoning ${{{\cal M}_{SAB}}}$ to guide the downstream tasks, and ${{{\cal S}_w}}$ estimates a refined flow that warps ${{\cal I}_B^{hc}}$ towards ${{\cal M}_{SAB}^{hc}}$. We concatenate ${\tilde{\cal I}_{SAB}^{hc}}$, ${\tilde{\cal I}_{SAB}^{ha}}$, and ${\tilde{\cal I}_{SAB}^{hr}}$ to yield final transfer result ${{\tilde{\cal I}_{SAB}}}$.
• Figure 3: Transfer parsing reasoning teaching. The pre-trained teacher model provides the feature and response knowledge to supervise the training of student model. The student model then advances its reasoning capability by learns the hard knowledge in the self-study phase.
• Figure 4: The heat maps of feature knowledge.
• Figure 5: Flow warping teaching. The student flow warping model comprises an initial aligning phase and a flow warping phase. The initial aligning phase learns an affine transformation to facilitate the position mapping, while the flow warping phase focuses on estimating a fine-grained flow to enable precise matching of the transferred garment with the target shape. The training of the student flow warping model is supervised by the shape information ${{\cal M}_{SAB}^{hc}}$ and the distillated content information ${\tilde{\cal I}_{TAB}^{hc}}$.