Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Block-wise LoRA: Revisiting Fine-grained LoRA for Effective Personalization and Stylization in Text-to-Image Generation

Likun Li, Haoqi Zeng, Changpeng Yang, Haozhe Jia, Di Xu

TL;DR

The paper tackles the challenge of achieving faithful personalization and stylization in text-to-image diffusion with limited resources. It introduces block-wise LoRA, a fine-grained PEFT approach that partitions the Stable Diffusion U-Net into blocks and selectively tunes them to separately encode identity and style. Empirical results show reduced conflicts between multiple LoRA adapters and improved subject fidelity and stylistic accuracy compared with traditional LoRA/LoCon baselines, with ablations identifying which blocks contribute most to identity versus style. The approach offers a practical, scalable path for personalized T2I generation and can be extended with ControlNet or orthogonal decomposition to further enhance control and efficiency.

Abstract

The objective of personalization and stylization in text-to-image is to instruct a pre-trained diffusion model to analyze new concepts introduced by users and incorporate them into expected styles. Recently, parameter-efficient fine-tuning (PEFT) approaches have been widely adopted to address this task and have greatly propelled the development of this field. Despite their popularity, existing efficient fine-tuning methods still struggle to achieve effective personalization and stylization in T2I generation. To address this issue, we propose block-wise Low-Rank Adaptation (LoRA) to perform fine-grained fine-tuning for different blocks of SD, which can generate images faithful to input prompts and target identity and also with desired style. Extensive experiments demonstrate the effectiveness of the proposed method.

Block-wise LoRA: Revisiting Fine-grained LoRA for Effective Personalization and Stylization in Text-to-Image Generation

TL;DR

The paper tackles the challenge of achieving faithful personalization and stylization in text-to-image diffusion with limited resources. It introduces block-wise LoRA, a fine-grained PEFT approach that partitions the Stable Diffusion U-Net into blocks and selectively tunes them to separately encode identity and style. Empirical results show reduced conflicts between multiple LoRA adapters and improved subject fidelity and stylistic accuracy compared with traditional LoRA/LoCon baselines, with ablations identifying which blocks contribute most to identity versus style. The approach offers a practical, scalable path for personalized T2I generation and can be extended with ControlNet or orthogonal decomposition to further enhance control and efficiency.

Abstract

The objective of personalization and stylization in text-to-image is to instruct a pre-trained diffusion model to analyze new concepts introduced by users and incorporate them into expected styles. Recently, parameter-efficient fine-tuning (PEFT) approaches have been widely adopted to address this task and have greatly propelled the development of this field. Despite their popularity, existing efficient fine-tuning methods still struggle to achieve effective personalization and stylization in T2I generation. To address this issue, we propose block-wise Low-Rank Adaptation (LoRA) to perform fine-grained fine-tuning for different blocks of SD, which can generate images faithful to input prompts and target identity and also with desired style. Extensive experiments demonstrate the effectiveness of the proposed method.
Paper Structure (11 sections, 2 equations, 4 figures)

This paper contains 11 sections, 2 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Parameter-efficient Fine-tuning (PEFT)
  4. T2I Personalization via Fine-tuning
  5. Preliminaries
  6. Method
  7. Block-wise Fine-grained Fine-tuning
  8. Experiments
  9. Implementation details
  10. Results
  11. Conclusion

Figures (4)

  • Figure 1: Block partitioning of the U-Net in SD.
  • Figure 2: Comparisons with LoRA hu2021lora. We collect the portraits of Anne Hathaway and cartoon images with a big head style as the training data for personalization and stylization, respectively. It can be observed that using LoRA fails to generate images with target style, while applying block-wise inference to LoRA struggles to generate corresponding anime images for Hathaway. Our method is capable to achieve expected personalization and stylization in the generated images. Applying our block-wise training to LoRA can achieve best generation results.
  • Figure 3: Comparisons with LoCon LyCORIS. We collect the portraits of Barack Obama and anime images with a specific style as the training data for personalization and stylization, respectively. Consistent with the comparison results in Figure \ref{['lora']}, applying our block-wise training to LoCon can still obtain best generation results.
  • Figure 4: Ablation studies of using block-wise LoCon training for personalization and stylization, as well as comparisons between applying block-wise LoCon training for different blocks of SD.