From Mannequin to Human: A Pose-Aware and Identity-Preserving Video Generation Framework for Lifelike Clothing Display

TL;DR

The paper tackles mannequin-based clothing displays by introducing mannequin-to-human (M2H) video generation, a cross-domain task that converts mannequin footage into photorealistic, identity-controlled human videos. It presents M2HVideo, which combines a dynamic pose-aware head encoder, a distribution-aware head-clothing adapter, and a pixel-space mirror loss within a latent diffusion framework guided by DDIM, to address head–body misalignment, identity drift, and loss of facial detail. Training leverages human videos as proxies, encoding clothing, pose, and identity signals to produce pose-consistent, identity-preserving renderings, with a one-step denoising strategy to recover high-frequency details. Extensive experiments on UBC fashion, ASOS, and MannequinVideos show superior clothing consistency, identity preservation, and temporal fidelity compared to state-of-the-art baselines, supported by ablation studies that validate each component. The work has practical impact for affordable, high-fidelity online fashion displays and offers a robust framework for cross-domain video synthesis with controllable identity.

Abstract

Mannequin-based clothing displays offer a cost-effective alternative to real-model showcases for online fashion presentation, but lack realism and expressive detail. To overcome this limitation, we introduce a new task called mannequin-to-human (M2H) video generation, which aims to synthesize identity-controllable, photorealistic human videos from footage of mannequins. We propose M2HVideo, a pose-aware and identity-preserving video generation framework that addresses two key challenges: the misalignment between head and body motion, and identity drift caused by temporal modeling. In particular, M2HVideo incorporates a dynamic pose-aware head encoder that fuses facial semantics with body pose to produce consistent identity embeddings across frames. To address the loss of fine facial details due to latent space compression, we introduce a mirror loss applied in pixel space through a denoising diffusion implicit model (DDIM)-based one-step denoising. Additionally, we design a distribution-aware adapter that aligns statistical distributions of identity and clothing features to enhance temporal coherence. Extensive experiments on the UBC fashion dataset, our self-constructed ASOS dataset, and the newly collected MannequinVideos dataset captured on-site demonstrate that M2HVideo achieves superior performance in terms of clothing consistency, identity preservation, and video fidelity in comparison to state-of-the-art methods.

Figures (7)

• Figure 1: Comparison of tasks related to M2H video generation. (a) Virtual try-on, which focuses on transferring clothing onto a target model while preserving identity. (b) Pose transfer, which aims to reproduce body posture while maintaining the fidelity of the appearance. (c) Mannequin-to-human video generation (ours), which generates photorealistic human frames that retain the clothing of the mannequin and adopt the features of the identity of the reference subject.
• Figure 2: Overview of the M2HVideo framework. (a) Training phase of M2HVideo using human videos. The body pose, identity image, and pose extracted from a human video are first aligned using the dynamic pose-aware head encoder. These features, along with clothing representations, are passed to a UNet to synthesize photorealistic human videos. The output is supervised by the loss function in both the latent and pixel spaces to ensure consistency and realism of the appearance. (b) Inference phase of M2HVideo with the mannequin video and identity image as input. Each frame of the mannequin video is processed to extract the clothing and body pose, which are then combined with an identity image and fed into our model to produce realistic human videos conditioned on the mannequin's clothing and the target identity.
• Figure 3: Statistics and visualizations of the datasets used in this study. (a) Distribution of video frame counts and data partitioning for the UBC fashion and ASOS datasets. (b) Clothing category distribution in the MannequinVideos dataset, illustrating the proportions of T-shirts, long-sleeved shirts, and dresses. (c) Data acquisition setup for the MannequinVideos dataset, showing the studio environment with a mannequin on a rotating platform and a smartphone used for capturing the video.
• Figure 4: Comparisons between our proposed M2HVideo and some baselines, namely, MagicPose chang2023magicpose, MimicMotion zhang2024mimicmotion, AnimateAnyone hu2024animate, StableVITON kim2024stableviton, and ControlNeXt peng2024ControlNeXt on the UBC fashion dataset.
• Figure 5: Comparisons between our proposed M2HVideo and the baselines, namely, MagicPose chang2023magicpose, MimicMotion zhang2024mimicmotion, AnimateAnyone hu2024animate, StableVITON kim2024stableviton, and ControlNeXt peng2024ControlNeXt on the ASOS dataset.