Hollowed Net for On-Device Personalization of Text-to-Image Diffusion Models

TL;DR

This paper presents an efficient LoRA-based personalization approach for on-device subject-driven generation, where pre-trained diffusion models are fine-tuned with user-specific data on resource-constrained devices.

Abstract

Recent advancements in text-to-image diffusion models have enabled the personalization of these models to generate custom images from textual prompts. This paper presents an efficient LoRA-based personalization approach for on-device subject-driven generation, where pre-trained diffusion models are fine-tuned with user-specific data on resource-constrained devices. Our method, termed Hollowed Net, enhances memory efficiency during fine-tuning by modifying the architecture of a diffusion U-Net to temporarily remove a fraction of its deep layers, creating a hollowed structure. This approach directly addresses on-device memory constraints and substantially reduces GPU memory requirements for training, in contrast to previous methods that primarily focus on minimizing training steps and reducing the number of parameters to update. Additionally, the personalized Hollowed Net can be transferred back into the original U-Net, enabling inference without additional memory overhead. Quantitative and qualitative analyses demonstrate that our approach not only reduces training memory to levels as low as those required for inference but also maintains or improves personalization performance compared to existing methods.

Figures (10)

• Figure 1: The LoRA personalization with Hollowed Net for resource-constrained environments. The input image is from the DreamBooth dataset ruiz2023dreambooth.
• Figure 2: Analysis of the LoRA weight change before and after personalization, per block of U-Net.
• Figure 3: The inference process with personalized LoRA parameters transferred from Hollowed Net to the original U-Net. The input image is from the DreamBooth dataset ruiz2023dreambooth.
• Figure 4: Qualitative generation results of Hollowed Net with different subjects and prompts. The upper half are the examples from the DreamBooth dataset ruiz2023dreambooth, and the lower half are the examples from the CustomConcept101 dataset kumari2023multi.
• Figure 5: Analysis of different fractions of hollowed layers. For all figures, the x-axis represents the fractions of layers removed from the pre-trained diffusion U-Net. The y-axis corresponds to the metric used for each figure.