Once Is Enough: Lightweight DiT-Based Video Virtual Try-On via One-Time Garment Appearance Injection

TL;DR

The paper tackles the high parameter and computation burden of Diffusion Transformer-based video virtual try-on by introducing OIE, a first-frame guided single-branch framework. OIE edits the initial frame with a pretrained image-based garment editor and propagates the edit through the video using a LoRA-finetuned DiT, aided by a lightweight background encoder and pose/mask guidance, while injecting garment appearance only once. The approach achieves state-of-the-art efficiency with minimal overhead—adding about $r ext{(low-rank)}$-based trainable parameters and negligible FLOPs—while maintaining or improving video quality on ViViD and ViViD-like benchmarks. Ablation studies confirm the value of pose guidance and mask guidance, and experiments demonstrate practical viability for resource-constrained settings.

Abstract

Video virtual try-on aims to replace the clothing of a person in a video with a target garment. Current dual-branch architectures have achieved significant success in diffusion models based on the U-Net; however, adapting them to diffusion models built upon the Diffusion Transformer remains challenging. Initially, introducing latent space features from the garment reference branch requires adding or modifying the backbone network, leading to a large number of trainable parameters. Subsequently, the latent space features of garments lack inherent temporal characteristics and thus require additional learning. To address these challenges, we propose a novel approach, OIE (Once is Enough), a virtual try-on strategy based on first-frame clothing replacement: specifically, we employ an image-based clothing transfer model to replace the clothing in the initial frame, and then, under the content control of the edited first frame, utilize pose and mask information to guide the temporal prior of the video generation model in synthesizing the remaining frames sequentially. Experiments show that our method achieves superior parameter efficiency and computational efficiency while still maintaining leading performance under these constraints.

Figures (4)

• Figure 1: Overall framework of OIE. OIE leverages a pre-trained image virtual try-on model to inject garment appearance, and further adapts the input configuration on a LoRA-finetuned model to achieve a low-parameter, structure-preserving video virtual try-on strategy.
• Figure 2: Comparison of Parameter Count. Dual-branch methods exhibit a training-to-inference parameter ratio lying along the dashed diagonal line (slope = 1), indicating nearly identical parameter counts during training and inference.
• Figure 3: Comparison of Computational Burden. Dashed lines denote baseline performance and dots represent models. OIE introduces negligible inference overhead compared to others.
• Figure 4: Visualization Comparison. OIE outperforms other models in garment detail control and pose consistency.