Any2AnyTryon: Leveraging Adaptive Position Embeddings for Versatile Virtual Clothing Tasks

TL;DR

Any2AnyTryon tackles data scarcity and task rigidity in clothing virtual try-on by casting VTON as a conditional generation problem conditioned on model and garment images and textual prompts. It introduces LAION-Garment, a large, diverse dataset, and Adaptive Position Embedding to align multiple input conditions in a diffusion-transformer backbone (FD: DiT/FLUX.1), enabling mask-free, multi-tasked VTON including model-driven, garment-driven, and layered edits. The architecture uses image-condition concatenation, clean latents, and LoRA fine-tuning, with a two-stage training scheme to cover broad tasks; quantitative and qualitative results show superior garment reconstruction, image quality (FID/KID), and controllability over state-of-the-art methods. This work advances practical, user-friendly virtual try-on with broader generalization to "in-the-wild" and shop settings and lays groundwork for flexible, instruction-driven fashion editing.

Abstract

Image-based virtual try-on (VTON) aims to generate a virtual try-on result by transferring an input garment onto a target person's image. However, the scarcity of paired garment-model data makes it challenging for existing methods to achieve high generalization and quality in VTON. Also, it limits the ability to generate mask-free try-ons. To tackle the data scarcity problem, approaches such as Stable Garment and MMTryon use a synthetic data strategy, effectively increasing the amount of paired data on the model side. However, existing methods are typically limited to performing specific try-on tasks and lack user-friendliness. To enhance the generalization and controllability of VTON generation, we propose Any2AnyTryon, which can generate try-on results based on different textual instructions and model garment images to meet various needs, eliminating the reliance on masks, poses, or other conditions. Specifically, we first construct the virtual try-on dataset LAION-Garment, the largest known open-source garment try-on dataset. Then, we introduce adaptive position embedding, which enables the model to generate satisfactory outfitted model images or garment images based on input images of different sizes and categories, significantly enhancing the generalization and controllability of VTON generation. In our experiments, we demonstrate the effectiveness of our Any2AnyTryon and compare it with existing methods. The results show that Any2AnyTryon enables flexible, controllable, and high-quality image-based virtual try-on generation. https://logn-2024.github.io/Any2anyTryonProjectPage

Figures (12)

• Figure 1: Overview of Any2AnyTryon.
• Figure 2: Qualitative comparison of garment reconstruction.
• Figure 3: Demonstration of multi garment flatten ability of our model with different prompt.
• Figure 4: Demonstration of garment-driven generation ability of our model with different image sizes.
• Figure 5: Qualitative comparison of virtual try-on on the VITONHD dataset.