Masked Extended Attention for Zero-Shot Virtual Try-On In The Wild

TL;DR

MaX4Zero tackles zero-shot virtual try-on in-the-wild, addressing unseen garments and unseen targets without fine-tuning. It introduces a two-stage pipeline: Initial Registration to warp the reference garment onto the target using deep-feature correspondence and fringe inpainting, and Consistent Inpainting via Masked Extended Attention to fuse reference texture details while preventing background leakage. The Masked Extended Attention mechanism expands attention across the two images but uses masks to control information flow, achieving better garment identity preservation and realistic results. Extensive experiments on DressCode and VITON-HD datasets, plus a user study, show competitive performance compared to supervised, fine-tuned baselines, with strong garment fidelity and image realism. The work broadens practical VTON deployment by reducing data requirements and enabling generalization to out-of-domain garments.

Abstract

Virtual Try-On (VTON) is a highly active line of research, with increasing demand. It aims to replace a piece of garment in an image with one from another, while preserving person and garment characteristics as well as image fidelity. Current literature takes a supervised approach for the task, impairing generalization and imposing heavy computation. In this paper, we present a novel zero-shot training-free method for inpainting a clothing garment by reference. Our approach employs the prior of a diffusion model with no additional training, fully leveraging its native generalization capabilities. The method employs extended attention to transfer image information from reference to target images, overcoming two significant challenges. We first initially warp the reference garment over the target human using deep features, alleviating "texture sticking". We then leverage the extended attention mechanism with careful masking, eliminating leakage of reference background and unwanted influence. Through a user study, qualitative, and quantitative comparison to state-of-the-art approaches, we demonstrate superior image quality and garment preservation compared unseen clothing pieces or human figures.

Figures (15)

• Figure 1: Overview of the proposed MaX4Zero method. Top: the Initial Registration stage, where the reference garment is warped to match the target person using extracted deep features from both images tang2023emergent. The remaining gaps between the target garment and warped one are filled by the Fringe Assignment module (see \ref{['fig:fringe']}). Bottom: the Consistent Inpainting stage, where utilizing the Masked Extended Attention mechanism for transferring the reference fine-details through stroke-based inpainting.
• Figure 2: Garment generation compared including or excluding the initial registration stage. This demonstrates the "texture sticking" phenomenon, where the spatial features of the transformed shirt are apparent in the resulting generation.
• Figure 3: Overview of the double-mask inpainting strategy. At each timestep, the dilated mask is used for noise prediction, but only the regions in the thin mask are actually taken to the next diffusion iteration.
• Figure 4: Qualitative comparison of MaX4Zero and competitors on in the wild target images. We compare against LaDI-VTON morelli2023ladivton and stableVTON kim2023stableviton, which are dedicated VTON approaches, and IP-adapter ye2023ip-adapter and Anydoor chen2024anydoor, which are personalized image editing and paint-by-reference methods respectively. As can be seen, garment identity is preserved better using our method for unseen garments.
• Figure 5: Results Gallery. Generated results by MaX4Zero. Best viewed when zoomed in.