CDM-QTA: Quantized Training Acceleration for Efficient LoRA Fine-Tuning of Diffusion Model
Jinming Lu, Minghao She, Wendong Mao, Zhongfeng Wang
TL;DR
The paper addresses the high resource cost of fine-tuning diffusion models for personalized concepts. It combines a LoRA-based fine-tuning scheme that updates only cross-attention projections via $Y = XW + XAB^{T}$ with a fully quantized training pipeline that uses INT8 precision and scales $S = X_{ ext{max}}/(2^{q-1}-1)$. A flexible hardware accelerator with a $64\times64$ systolic PE array supports WS and OS dataflows to maximize utilization during irregular LoRA cross-attention computations. Results show up to $1.81\times$ training speedup and $5.50\times$ energy efficiency improvements over baselines, enabling practical edge deployment of customized diffusion models.
Abstract
Fine-tuning large diffusion models for custom applications demands substantial power and time, which poses significant challenges for efficient implementation on mobile devices. In this paper, we develop a novel training accelerator specifically for Low-Rank Adaptation (LoRA) of diffusion models, aiming to streamline the process and reduce computational complexity. By leveraging a fully quantized training scheme for LoRA fine-tuning, we achieve substantial reductions in memory usage and power consumption while maintaining high model fidelity. The proposed accelerator features flexible dataflow, enabling high utilization for irregular and variable tensor shapes during the LoRA process. Experimental results show up to 1.81x training speedup and 5.50x energy efficiency improvements compared to the baseline, with minimal impact on image generation quality.