Pursuing Temporal-Consistent Video Virtual Try-On via Dynamic Pose Interaction

TL;DR

DPIDM addresses temporal inconsistency in video virtual try-on by explicitly modeling intra-frame and long-term pose interactions between the wearer and garment using a skeleton-based pose adapter and a hierarchical pose-aware diffusion backbone. It couples a dual-branch diffusion system (Main U-Net and Garment U-Net) with pose-guided attention modules—PASA, TSA, CA, and PATA—and a temporal regularized attention loss to boost temporal coherence. Across VVT, ViViD, and VITON-HD, it achieves state-of-the-art VFID and image-quality metrics, demonstrating improved garment fidelity and motion consistency. The results offer a practical pathway for diffusion-based VVTON that leverages pose priors to handle complex garment-body interactions.

Abstract

Video virtual try-on aims to seamlessly dress a subject in a video with a specific garment. The primary challenge involves preserving the visual authenticity of the garment while dynamically adapting to the pose and physique of the subject. While existing methods have predominantly focused on image-based virtual try-on, extending these techniques directly to videos often results in temporal inconsistencies. Most current video virtual try-on approaches alleviate this challenge by incorporating temporal modules, yet still overlook the critical spatiotemporal pose interactions between human and garment. Effective pose interactions in videos should not only consider spatial alignment between human and garment poses in each frame but also account for the temporal dynamics of human poses throughout the entire video. With such motivation, we propose a new framework, namely Dynamic Pose Interaction Diffusion Models (DPIDM), to leverage diffusion models to delve into dynamic pose interactions for video virtual try-on. Technically, DPIDM introduces a skeleton-based pose adapter to integrate synchronized human and garment poses into the denoising network. A hierarchical attention module is then exquisitely designed to model intra-frame human-garment pose interactions and long-term human pose dynamics across frames through pose-aware spatial and temporal attention mechanisms. Moreover, DPIDM capitalizes on a temporal regularized attention loss between consecutive frames to enhance temporal consistency. Extensive experiments conducted on VITON-HD, VVT and ViViD datasets demonstrate the superiority of our DPIDM against the baseline methods. Notably, DPIDM achieves VFID score of 0.506 on VVT dataset, leading to 60.5% improvement over the state-of-the-art GPD-VVTO approach.

Figures (5)

• Figure 1: Given a garment and a person video, DPIDM generates a lifelike video that preserves the visual authenticity of the garment while adapting to the individual's pose and physique.
• Figure 2: (a) Overall architecture of DPIDM. DPIDM emplys a dual-branch architecture. The main U-Net processes a concatenated input comprising the noisy latent of the video, the latent of the cloth-agnostic video, and the cloth-agnostic mask sequence. The garment U-Net extracts fine-grained garment features, which are subsequently integrated into the main U-Net. The pose estimator is utilized to extract aligned human and garment poses, which are then fed into the attention modules of the main U-Net to guide the diffusion process. The VAE is not shown for clarity. (b) Detailed illustration of the proposed pose-aware attention module within the main U-Net. The module comprises pose-aware spatial attention, temporal-shift attention, cross-attention, and pose-aware temporal attention. The pose embeddings are seamlessly integrated into the attention module through a specialized pose adapter. (better viewed in color)
• Figure 3: Visualization of predicted human and garment poses.
• Figure 4: Qualitative comparison on the ViViD dataset. Our DPIDM excels in seamlessly integrating the garment with the wearer and maintaining the visual integrity of the garment.
• Figure 5: Qualitative comparison on the ViViD dataset. Our DPIDM maintains temporal consistency even during substantial movements.