Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mobile Fitting Room: On-device Virtual Try-on via Diffusion Models

Justin Blalock, David Munechika, Harsha Karanth, Alec Helbling, Pratham Mehta, Seongmin Lee, Duen Horng Chau

TL;DR

This work tackles privacy-aware virtual try-on by delivering a fully on-device diffusion-based pipeline for garment placement, controlled generation, and user customization. It combines DreamBooth-style fine-tuning for garment specificity with CoreML-based compression (including 6-bit palettization and U-Net splitting) and inpainting-driven region control to enable interactive, offline mobile experiences. The approach preserves user privacy, supports personalized interactions, and demonstrates practical deployment potential through a retail-facing scenario, with planned evaluations and app-store deployment. The key contribution is a cohesive, mobile-first diffusion framework that balances quality, efficiency, and usability for fashion e-commerce on consumer devices.

Abstract

The growing digital landscape of fashion e-commerce calls for interactive and user-friendly interfaces for virtually trying on clothes. Traditional try-on methods grapple with challenges in adapting to diverse backgrounds, poses, and subjects. While newer methods, utilizing the recent advances of diffusion models, have achieved higher-quality image generation, the human-centered dimensions of mobile interface delivery and privacy concerns remain largely unexplored. We present Mobile Fitting Room, the first on-device diffusion-based virtual try-on system. To address multiple inter-related technical challenges such as high-quality garment placement and model compression for mobile devices, we present a novel technical pipeline and an interface design that enables privacy preservation and user customization. A usage scenario highlights how our tool can provide a seamless, interactive virtual try-on experience for customers and provide a valuable service for fashion e-commerce businesses.

Mobile Fitting Room: On-device Virtual Try-on via Diffusion Models

TL;DR

This work tackles privacy-aware virtual try-on by delivering a fully on-device diffusion-based pipeline for garment placement, controlled generation, and user customization. It combines DreamBooth-style fine-tuning for garment specificity with CoreML-based compression (including 6-bit palettization and U-Net splitting) and inpainting-driven region control to enable interactive, offline mobile experiences. The approach preserves user privacy, supports personalized interactions, and demonstrates practical deployment potential through a retail-facing scenario, with planned evaluations and app-store deployment. The key contribution is a cohesive, mobile-first diffusion framework that balances quality, efficiency, and usability for fashion e-commerce on consumer devices.

Abstract

The growing digital landscape of fashion e-commerce calls for interactive and user-friendly interfaces for virtually trying on clothes. Traditional try-on methods grapple with challenges in adapting to diverse backgrounds, poses, and subjects. While newer methods, utilizing the recent advances of diffusion models, have achieved higher-quality image generation, the human-centered dimensions of mobile interface delivery and privacy concerns remain largely unexplored. We present Mobile Fitting Room, the first on-device diffusion-based virtual try-on system. To address multiple inter-related technical challenges such as high-quality garment placement and model compression for mobile devices, we present a novel technical pipeline and an interface design that enables privacy preservation and user customization. A usage scenario highlights how our tool can provide a seamless, interactive virtual try-on experience for customers and provide a valuable service for fashion e-commerce businesses.
Paper Structure (16 sections, 2 figures)

This paper contains 16 sections, 2 figures.

Table of Contents

  1. Introduction
  2. User Interface
  3. Overview
  4. Garments Window
  5. Personalization Window
  6. Result Page
  7. Techniques for On-Device Controlled Diffusion
  8. Model Fine-Tuning
  9. Model Conversion and Inference
  10. Inpainting
  11. Preliminary Result: Virtual Try-On Usage Scenario for Retail Business
  12. Ongoing Work
  13. Model Improvements
  14. Planned Evaluation
  15. Deployment
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Samples of generated images from Mobile Fitting Room. The user inputs an image and draws a mask around the area that they want replaced.
  • Figure 2: The pipeline for Mobile Fitting Room allows for an on-device privacy-preserving model whose flexibility allows for user personalization through interaction. (A) Fine-tuning generates a model that allows us to summon a specific garment with a unique identifier. (B) We then compress the model so that it can meet the storage and efficiency demands of a mobile device. (C) Mobile Fitting Room allows users to specify where they want a garment to be placed by using ControlNet inpainting.