The devil is in the details: Enhancing Video Virtual Try-On via Keyframe-Driven Details Injection

TL;DR

This work tackles the limitations of diffusion transformer–based video virtual try-on by introducing KeyTailor, a lightweight keyframe-driven detail injection framework that enhances garment dynamics and background fidelity without altering the DiT backbone. It relies on instruction-guided keyframe sampling and two dedicated modules—Garment Dynamics Details Enhancement and Collaborative Background Details Optimization—to distill fine-grained garment variations and preserve background integrity, with a LoRA-based parameter-efficient adaptation. To address data scarcity, the authors curate ViT-HD, a large-scale high-definition dataset with 15,070 samples at 810×1080. Empirical results on ViT-HD, ViViD, and image-based VTON benchmarks show KeyTailor outperforms state-of-the-art baselines in both garment fidelity and background consistency, while maintaining competitive computational efficiency. The work presents a scalable approach for high-quality VVT and provides a valuable dataset to advance future research in this domain.

Abstract

Although diffusion transformer (DiT)-based video virtual try-on (VVT) has made significant progress in synthesizing realistic videos, existing methods still struggle to capture fine-grained garment dynamics and preserve background integrity across video frames. They also incur high computational costs due to additional interaction modules introduced into DiTs, while the limited scale and quality of existing public datasets also restrict model generalization and effective training. To address these challenges, we propose a novel framework, KeyTailor, along with a large-scale, high-definition dataset, ViT-HD. The core idea of KeyTailor is a keyframe-driven details injection strategy, motivated by the fact that keyframes inherently contain both foreground dynamics and background consistency. Specifically, KeyTailor adopts an instruction-guided keyframe sampling strategy to filter informative frames from the input video. Subsequently,two tailored keyframe-driven modules, the garment details enhancement module and the collaborative background optimization module, are employed to distill garment dynamics into garment-related latents and to optimize the integrity of background latents, both guided by keyframes.These enriched details are then injected into standard DiT blocks together with pose, mask, and noise latents, enabling efficient and realistic try-on video synthesis. This design ensures consistency without explicitly modifying the DiT architecture, while simultaneously avoiding additional complexity. In addition, our dataset ViT-HD comprises 15, 070 high-quality video samples at a resolution of 810*1080, covering diverse garments. Extensive experiments demonstrate that KeyTailor outperforms state-of-the-art baselines in terms of garment fidelity and background integrity across both dynamic and static scenarios.

Figures (13)

• Figure 1: (a) Comparison of garment details; (b) Comparison of background details; (c) Comparison of parameters and efficienty.
• Figure 2: Dataset overview.
• Figure 3: Overall framework of KeyTailor. KeyTailor takes as input a reference garment image $I_{\textit{ref}}$, a source video $V_{\textit{in}}$, its corresponding agnostic video $V_{\textit{agn}}$, agnostic masks $M_{\textit{agn}}$, and pose representations $P$. These inputs are encoded into garment-related latents $L_g$, background-related latents $L_{\textit{bg}}$, pose latents $L_p$, and resized masks $L_m$. Specifically, garment-related latents are generated by the GDDE module, background-related latents by the CBDO module, and pose latents by a trainable pose guider. Subsequently, all these latents, together with noise latents, are injected into $N$ DiT blocks to produce the final try-on video tokens, which are then decoded by a VAE-based video decoder to synthesize the output video.
• Figure 4: Qualitative comparison of video virtual try-on results on the ViViD dataset (1st column), our ViT-HD dataset (2nd column), and in-the-wild scenarios (3rd column). Our KeyTailor restores fine-grained garment details while preserving background integrity.
• Figure 5: Qualitative results and comparisons in person-to-video garment transfer scenarios. Our method combines background, person, and garment more naturally in complex scenarios.