A LoRA is Worth a Thousand Pictures

TL;DR

This work demonstrates that Low Rank Adaptation (LoRA) weights, trained on small art datasets, can function as standalone style embeddings for artist-style analysis. By extracting a compact representation via PCA from concatenated LoRA components and applying a calibration step to mitigate projection drift, the authors show that LoRA-based embeddings yield well-separated style clusters that align with art-historical knowledge and outperform traditional image-based features in clustering and retrieval tasks. The approach enables direct retrieval of LoRA models by style without requiring image generation or access to original training data, offering practical benefits for online LoRA communities and implications for zero-shot fine-tuning and model attribution. The method also reveals which LoRA sub-networks contribute most to style representation, with feedforward components delivering strong performance at reduced dimensionality. Overall, the work provides a new, efficient way to analyze, organize, and attribute customized diffusion models at scale, with potential applications in attribution, compensation, and trustworthy sharing of stylized AI resources.

Abstract

Recent advances in diffusion models and parameter-efficient fine-tuning (PEFT) have made text-to-image generation and customization widely accessible, with Low Rank Adaptation (LoRA) able to replicate an artist's style or subject using minimal data and computation. In this paper, we examine the relationship between LoRA weights and artistic styles, demonstrating that LoRA weights alone can serve as an effective descriptor of style, without the need for additional image generation or knowledge of the original training set. Our findings show that LoRA weights yield better performance in clustering of artistic styles compared to traditional pre-trained features, such as CLIP and DINO, with strong structural similarities between LoRA-based and conventional image-based embeddings observed both qualitatively and quantitatively. We identify various retrieval scenarios for the growing collection of customized models and show that our approach enables more accurate retrieval in real-world settings where knowledge of the training images is unavailable and additional generation is required. We conclude with a discussion on potential future applications, such as zero-shot LoRA fine-tuning and model attribution.

Figures (14)

• Figure 1: (a) Retrieving customized models based on image features requires additional image generation (red arrows), which depends on the choice of hyperparameters. (b) Direct retrieval of customized models using the model weights themselves is accurate and avoids extra costs as well as the need for choosing inference hyperparameters, including noise image seeds and prompts.
• Figure 1: Example artworks used for LoRA training ("Original") and images generated by the resulting LoRAs prompted by the automatic captions of the original images ("Generated"). These LoRAs tend to memorize the training images, especially when the training image set is small under the Test[Diff] setup.
• Figure 2: (a) We construct an embedding of style-customized LoRAs via PCA. (b) This embedding enables applications such as retrieval using a query LoRA trained on unknown images, avoiding the need for additional image generation.
• Figure 2: Example retrieval results. Artists' names are colored by the corresponding genres: art nouveau, baroque, post impressionism, renaissance, and ukiyo-e. Representative images of the retrieved results are framed in green for correct answers and red for incorrect ones.
• Figure 3: (a) Visualization of our artistic style LoRA embedding under the first two PCs. We use the same genre convex hulls as Fig. \ref{['fig:teaser']} and plot artist subsets across subfigures to avoid crowded visuals. (b) Samples are embedded in alignment with art historical knowledge; for example, a teacher and apprentice pair, known for sharing a similar style, are close in the embedding space.