An Identity-Preserved Framework for Human Motion Transfer

TL;DR

The proposed IDPres method surpasses existing state-of-the-art techniques in terms of reconstruction accuracy, realistic motion, and identity preservation, and is the first to introduce a new quantitative metric called Identity Score (ID-Score), motivated by the success of gait recognition methods in capturing identity information.

Abstract

Human motion transfer (HMT) aims to generate a video clip for the target subject by imitating the source subject's motion. Although previous methods have achieved good results in synthesizing good-quality videos, they lose sight of individualized motion information from the source and target motions, which is significant for the realism of the motion in the generated video. To address this problem, we propose a novel identity-preserved HMT network, termed \textit{IDPres}. This network is a skeleton-based approach that uniquely incorporates the target's individualized motion and skeleton information to augment identity representations. This integration significantly enhances the realism of movements in the generated videos. Our method focuses on the fine-grained disentanglement and synthesis of motion. To improve the representation learning capability in latent space and facilitate the training of \textit{IDPres}, we introduce three training schemes. These schemes enable \textit{IDPres} to concurrently disentangle different representations and accurately control them, ensuring the synthesis of ideal motions. To evaluate the proportion of individualized motion information in the generated video, we are the first to introduce a new quantitative metric called Identity Score (\textit{ID-Score}), motivated by the success of gait recognition methods in capturing identity information. Moreover, we collect an identity-motion paired dataset, $Dancer101$, consisting of solo-dance videos of 101 subjects from the public domain, providing a benchmark to prompt the development of HMT methods. Extensive experiments demonstrate that the proposed \textit{IDPres} method surpasses existing state-of-the-art techniques in terms of reconstruction accuracy, realistic motion, and identity preservation.

Figures (15)

• Figure 1: An example of an unnatural case is presented by TransMoMo yang2020transmomo. The motion from the source video (the girl in white) is transferred to the target videos (the man in blue, and the girl in red). The synthesized videos from previous methods show a limitation: regardless of the subjects' identities, they tend to exhibit similar postures. In contrast, our proposed method considers individualized information and generates the natural posture of each individual.
• Figure 2: The overview of the proposed IDPres. (a) The overview of the training stage. Our training stage contains specific data sampling (Section \ref{['subsubsec:data_sampling']}) and three training schemes: self-reconstruction training (Section \ref{['subsubsec:self_reconstruction']}), overlap attribute training (Section \ref{['subsubsec:overlap']}), and non-overlap attribute training (Section \ref{['subsubsec:nonoverlap']}). (b) The overview of the inference stage. This stage takes two videos with different subjects and movements as input.
• Figure 3: The overview of (a) disentanglement block, $\mathbf{E}$ and (b) identity-broadcasted upsampling blocks, IBup. In subfigure (a), 1D-Conv s=$i$ represents the temporal convolutional layer with a stride of $i$ and a LeakyReLU activation function, IN means instance normalization, $T$ is the number of frames, $C_{k} (k \in \{MC, IM, HS\}$) is the number of channels of $k$, GAP and GMP are temporal-level global average pooling and global max pooling respectively. FC means the full-connection layer. In subfigure (b), FC is used to learn the mean and variance of $IM$, $z$ is a Gaussian noise to improve the diversity of generated motions.
• Figure 4: The different data pairs for training. Real datasets typically lack pairs of samples that share the same $HS$ while differing in the other two attributes ($MC$, $IM$). To overcome this limitation, we employ a technique called limb scaler operation yang2020transmomo. This method allows us to artificially create a new sample, denoted as $M_r^{0'}$, which maintains consistency in two attributes ($MC$ and $IM$) but exhibits a different $HS$.
• Figure 5: The self-reconstruction pipeline.