3DV-TON: Textured 3D-Guided Consistent Video Try-on via Diffusion Models

TL;DR

3DV-TON tackles the challenge of generating temporally coherent video try-on results by introducing animatable textured 3D guidance that constrains garment texture motion across frames. The method builds a textured 3D mesh from SMPL/SMPL-X and animates it in sync with the input video, integrating this guidance into a diffusion-based generator with two reference conditions. A robust rectangular masking strategy prevents garment leakage, and a high-resolution HR-VVT benchmark enables rigorous evaluation. Empirical results on ViViD and HR-VVT demonstrate improved texture fidelity and temporal coherence over state-of-the-art approaches, with competitive reconstruction speed on standard hardware.

Abstract

Video try-on replaces clothing in videos with target garments. Existing methods struggle to generate high-quality and temporally consistent results when handling complex clothing patterns and diverse body poses. We present 3DV-TON, a novel diffusion-based framework for generating high-fidelity and temporally consistent video try-on results. Our approach employs generated animatable textured 3D meshes as explicit frame-level guidance, alleviating the issue of models over-focusing on appearance fidelity at the expanse of motion coherence. This is achieved by enabling direct reference to consistent garment texture movements throughout video sequences. The proposed method features an adaptive pipeline for generating dynamic 3D guidance: (1) selecting a keyframe for initial 2D image try-on, followed by (2) reconstructing and animating a textured 3D mesh synchronized with original video poses. We further introduce a robust rectangular masking strategy that successfully mitigates artifact propagation caused by leaking clothing information during dynamic human and garment movements. To advance video try-on research, we introduce HR-VVT, a high-resolution benchmark dataset containing 130 videos with diverse clothing types and scenarios. Quantitative and qualitative results demonstrate our superior performance over existing methods. The project page is at this link https://2y7c3.github.io/3DV-TON/

Figures (20)

• Figure 1: Try-on videos generated by 3DV-TON. Our method can handle various types of clothing and body poses, while accurately restoring clothing details and maintaining consistent texture motion.
• Figure 2: Textured 3D guidance. We construct the textured 3D guidance based on image try-on results, then animate the mesh after pasting the texture, providing a consistent texture motion reference on the appearance level.
• Figure 3: The overview of 3DV-TON. Given a video, we first use our 3D guidance pipeline to select a frame $I$ adaptively, then reconstruct a textured 3D guidance and animate it align with the original video, i.e.$V$. We employ a guidance feature extractor for the clothing image $C$ and the try-on images $C_t$, and perform feature fusion using the self-attentions in the denoising UNet.
• Figure 4: Qualitative comparison for dress try-on on the ViViD dataset.
• Figure 5: Qualitative comparison for upper garment try-on on the ViViD dataset.