IPVTON: Image-based 3D Virtual Try-on with Image Prompt Adapter

TL;DR

IPVTON tackles the challenge of generating multi-view 3D virtual try-on from a single person image and a garment image by leveraging Score Distillation Sampling (SDS) with an Image Prompt Adapter (IP-Adapter) to inject garment features into a diffusion prior. It employs a two-stage framework that first optimizes a hybrid 3D representation initialized from SMPL-X to capture geometry using a normal map guided SDS loss and a pseudo silhouette loss $L_{PSL}$, then optimizes texture using garment-image prompts with masking to preserve non-try-on regions. The method uses mask-guided image prompt embeddings and a ControlNet-guided pseudo silhouette to constrain the garment contours, achieving accurate geometry and high-quality textures; the results outperform prior 2D/3D try-on baselines in both qualitative and CLIP-based quantitative metrics. This yields realistic, view-consistent 3D try-on with practical implications for e-commerce and AR applications, while maintaining source identity across views.

Abstract

Given a pair of images depicting a person and a garment separately, image-based 3D virtual try-on methods aim to reconstruct a 3D human model that realistically portrays the person wearing the desired garment. In this paper, we present IPVTON, a novel image-based 3D virtual try-on framework. IPVTON employs score distillation sampling with image prompts to optimize a hybrid 3D human representation, integrating target garment features into diffusion priors through an image prompt adapter. To avoid interference with non-target areas, we leverage mask-guided image prompt embeddings to focus the image features on the try-on regions. Moreover, we impose geometric constraints on the 3D model with a pseudo silhouette generated by ControlNet, ensuring that the clothed 3D human model retains the shape of the source identity while accurately wearing the target garments. Extensive qualitative and quantitative experiments demonstrate that IPVTON outperforms previous methods in image-based 3D virtual try-on tasks, excelling in both geometry and texture.

Figures (6)

• Figure 1: Compared to 2D virtual try-on kim2024stableviton with its fixed viewpoint and 3D virtual try-on li2024diffavatar that require complex processes, IPVTON can generate 3D try-on results from just a human image and a garment image.
• Figure 2: 3D Try-on results. Given a human image, a garment image and a text prompt, IPVTON can generate realistic 3D human models with the desired garment shapes and textures while preserving the source identity.
• Figure 3: Overview of IPVTON. Given a human image $\mathcal{H}_I$, we first construct a DMTet-based 3d representation initialized with SMPL-X to model the human, with its geometry and texture generated through $\Omega_g$ and $\Omega_c$, respectively. During geometry optimization, the rendered human normal map $I^n$ is encoded into the diffusion model $\hat{\epsilon}_{ip}$ and, along with $y_n$ and $m$, is used to compute $\mathcal{L}_{SDS}^{norm}$. $y_n$ is the normal image prompt embedding encoded from $\mathcal{H}_g^n$ via $\mathcal{E}_{ip}$, and $m$ is a mask covering the try-on region, derived from $\mathcal{H}_I'$. During texture optimization, the rendered human image $I^r$ is encoded into $\hat{\epsilon}_{ip}$ and along with $y_r,y$ and $m$, is used to compute $\mathcal{L}_{SDS}^{tex}$. $y$ is the text prompt embedding encoded from the target texts via $\mathcal{E}_{t}$, and $y_r$ is the image prompt embedding encoded from $\mathcal{H}_g$ via $\mathcal{E}_{ip}$. $\odot$ denotes pixel-wise multiplication.
• Figure 4: Qualitative comparisons. Our IPVTON is able to generate realistic 3D try-on results with high-quality textures, viewable from multiple angles.
• Figure 5: Ablation study for geometry optimization.$\texttt{'}$mIPA$\texttt{'}$ denotes mask-guided image prompt embeddings.