Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Air to Wear: Personalized 3D Digital Fashion with AR/VR Immersive 3D Sketching

Ying Zang, Yuanqi Hu, Xinyu Chen, Yuxia Xu, Suhui Wang, Chunan Yu, Lanyun Zhu, Deyi Ji, Xin Xu, Tianrun Chen

TL;DR

This work tackles the barrier to democratized 3D fashion design by enabling novices to create personalized garments from immersive 3D VR sketches. It introduces a conditional diffusion framework, trained in three stages with a shared 3D point-cloud latent space and an adaptive curriculum, to transform rough sketches into high-fidelity 3D garments. To address data scarcity, the KO3DClothes dataset of 969 paired 3D garments and VR sketches is released, and the training strategy couples a point-cloud encoder with a sketch encoder for robust conditioning. Experiments and user studies show competitive shape fidelity and superior usability compared with baselines, highlighting the approach’s potential for metaverse visualization, virtual try-on, and avatar customization on next-gen AR/VR platforms.

Abstract

In the era of immersive consumer electronics, such as AR/VR headsets and smart devices, people increasingly seek ways to express their identity through virtual fashion. However, existing 3D garment design tools remain inaccessible to everyday users due to steep technical barriers and limited data. In this work, we introduce a 3D sketch-driven 3D garment generation framework that empowers ordinary users - even those without design experience - to create high-quality digital clothing through simple 3D sketches in AR/VR environments. By combining a conditional diffusion model, a sketch encoder trained in a shared latent space, and an adaptive curriculum learning strategy, our system interprets imprecise, free-hand input and produces realistic, personalized garments. To address the scarcity of training data, we also introduce KO3DClothes, a new dataset of paired 3D garments and user-created sketches. Extensive experiments and user studies confirm that our method significantly outperforms existing baselines in both fidelity and usability, demonstrating its promise for democratized fashion design on next-generation consumer platforms.

From Air to Wear: Personalized 3D Digital Fashion with AR/VR Immersive 3D Sketching

TL;DR

This work tackles the barrier to democratized 3D fashion design by enabling novices to create personalized garments from immersive 3D VR sketches. It introduces a conditional diffusion framework, trained in three stages with a shared 3D point-cloud latent space and an adaptive curriculum, to transform rough sketches into high-fidelity 3D garments. To address data scarcity, the KO3DClothes dataset of 969 paired 3D garments and VR sketches is released, and the training strategy couples a point-cloud encoder with a sketch encoder for robust conditioning. Experiments and user studies show competitive shape fidelity and superior usability compared with baselines, highlighting the approach’s potential for metaverse visualization, virtual try-on, and avatar customization on next-gen AR/VR platforms.

Abstract

In the era of immersive consumer electronics, such as AR/VR headsets and smart devices, people increasingly seek ways to express their identity through virtual fashion. However, existing 3D garment design tools remain inaccessible to everyday users due to steep technical barriers and limited data. In this work, we introduce a 3D sketch-driven 3D garment generation framework that empowers ordinary users - even those without design experience - to create high-quality digital clothing through simple 3D sketches in AR/VR environments. By combining a conditional diffusion model, a sketch encoder trained in a shared latent space, and an adaptive curriculum learning strategy, our system interprets imprecise, free-hand input and produces realistic, personalized garments. To address the scarcity of training data, we also introduce KO3DClothes, a new dataset of paired 3D garments and user-created sketches. Extensive experiments and user studies confirm that our method significantly outperforms existing baselines in both fidelity and usability, demonstrating its promise for democratized fashion design on next-generation consumer platforms.
Paper Structure (27 sections, 5 equations, 4 figures, 4 tables)

This paper contains 27 sections, 5 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. 3D Garment Design with Deep Learning Algorithm.
  4. 3D Model Generation with Generative Models
  5. 2D Sketches or 3D Sketches
  6. KO3DClothes Dataset
  7. Method
  8. Preliminary: Conditional 3D Diffusion Model
  9. Principle
  10. Coarse-to-Fine Diffusion Network
  11. Learning the Conditional Distribution
  12. The Proposed Multi-stage Training Steategy
  13. Stage 1: Pre-Training the 3D Priors
  14. Stage 2: 3D Sketch Mapping
  15. Stage 3: Joint Fine-Tuning
  16. ...and 12 more sections

Figures (4)

  • Figure 2: The visualization of hand-drawn 3D sketch samples from KO3DClothes dataset.
  • Figure 3: The Overview of Deep3DVRSketch+. (a) Pre-training a conditional diffusion model by sampling ground truth (GT) point clouds. (b) Fine-tuning the sketch encoder to project sketches onto the diffusion manifold. (c) Curriculum learning leverages a limited set of sketch-shape pairs.
  • Figure 4: Comparison with the existing state-of-the-art methods.
  • Figure 5: Qualitative Evaluation for Ablation Studies.