Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation

TL;DR

This paper introduces LyCORIS (Lora beYond Conventional methods, Other Rank adaptation Implementations for Stable diffusion) [https://github.com/KohakuBlueleaf/LyCORIS], an open-source library that offers a wide selection of fine-tuning methodologies for St stable Diffusion, and presents a thorough framework for the systematic assessment of varied fine- Tuning techniques.

Abstract

Text-to-image generative models have garnered immense attention for their ability to produce high-fidelity images from text prompts. Among these, Stable Diffusion distinguishes itself as a leading open-source model in this fast-growing field. However, the intricacies of fine-tuning these models pose multiple challenges from new methodology integration to systematic evaluation. Addressing these issues, this paper introduces LyCORIS (Lora beYond Conventional methods, Other Rank adaptation Implementations for Stable diffusion) [https://github.com/KohakuBlueleaf/LyCORIS], an open-source library that offers a wide selection of fine-tuning methodologies for Stable Diffusion. Furthermore, we present a thorough framework for the systematic assessment of varied fine-tuning techniques. This framework employs a diverse suite of metrics and delves into multiple facets of fine-tuning, including hyperparameter adjustments and the evaluation with different prompt types across various concept categories. Through this comprehensive approach, our work provides essential insights into the nuanced effects of fine-tuning parameters, bridging the gap between state-of-the-art research and practical application.

Figures (41)

• Figure 1: This figure shows the structure of the proposed Loha and Lokr modules implemented in .
• Figure 2: SHAP beeswarm charts and scatter plots for analyzing the impact of change in different algorithm components. In the beeswarm plots, is in blue, is in purple, is in purple red, and is in red. Model capacity is adjusted by either increasing dimension (for or ) or decreasing factor (for ). In the scatter plots, SCD indicates that we use squared centroid distance to measure image similarity. This removes the implicit penalization towards more diverse image sets in the computation of average cosine similarity (see \ref{['apx:metrics']} for details). We believe it is thus more suitable when we are interested in the trade-off between fidelity and diversity. The error bars in the scatter plots represent standard errors of the metric values across random seeds and classes.
• Figure 3: Qualitative comparison of checkpoints trained with different configurations. Samples of the top row are generated using only concept descriptors while samples of the bottom row are generated with the two prompts "[$V_{\text{castle}}$] scene stands against a backdrop of snow-capped mountains" and "[$V_{\text{castle}}$] scene surrounded by a lush, vibrant forest". The number of training epochs is chosen according to the concept category.
• Figure 4: This figure shows how to represent as two or three linear layers (depending on whether we perform the additional low-rank decomposition or not).
• Figure 5: Pearson correlation for metrics computed using different encoders and resizing methods, evaluated on images generated from different types of prompts (best viewed when zoomed in).

Theorems & Definitions (3)

• theorem 1
• proof
• remark 1