LoRAverse: A Submodular Framework to Retrieve Diverse Adapters for Diffusion Models

TL;DR

LoRAverse reframes the challenge of selecting diverse, relevant LoRA adapters from vast diffusion-model libraries as a monotone submodular maximization problem. It introduces a two-stage architecture with a concept extractor to parse prompts into concepts and a submodular retriever that clusters adapters and optimizes relevance and diversity per concept, augmented by an adapter safety checker. Empirical results show notable gains in output diversity across domains while maintaining competitive text–image alignment, outperforming Stylus and SD v1.5 on diversity metrics and user judgments. The approach is backbone-agnostic and scalable, offering safer, more expressive personalization of diffusion models through principled adapter selection.

Abstract

Low-rank Adaptation (LoRA) models have revolutionized the personalization of pre-trained diffusion models by enabling fine-tuning through low-rank, factorized weight matrices specifically optimized for attention layers. These models facilitate the generation of highly customized content across a variety of objects, individuals, and artistic styles without the need for extensive retraining. Despite the availability of over 100K LoRA adapters on platforms like Civit.ai, users often face challenges in navigating, selecting, and effectively utilizing the most suitable adapters due to their sheer volume, diversity, and lack of structured organization. This paper addresses the problem of selecting the most relevant and diverse LoRA models from this vast database by framing the task as a combinatorial optimization problem and proposing a novel submodular framework. Our quantitative and qualitative experiments demonstrate that our method generates diverse outputs across a wide range of domains.

Figures (14)

• Figure 1: Diverse Adapter Selection. LoRAverse enhances image diversity by employing a submodular algorithm to select a diverse and representative set of LoRA adapters. This approach begins by clustering the adapters based on their semantic meanings. From these clusters, the algorithm selects models that not only maximize diversity but also maintain strong alignment with the user-provided prompt, ensuring both variety and relevance in the generated images.
• Figure 2: Architecture of LoRAverse. LoRAverse composed of two main modules: concept extractor and submodular retriever. The concept extractor processes the user prompt to identify concepts (keywords). These concepts guide to the submodular retriever, which selects a diverse and relevant subset of LoRA adapters by clustering similar adapters per concept and applying submodular optimization. Additionally, a safety-checking mechanism is integrated to filter out adapters containing offensive or inappropriate content.
• Figure 3: Qualitative Comparison. LoRAverse demonstrates a higher diversity compared to image sets generated by Stylus and SD v1.5.
• Figure 4: Impact of CFG values. The top shows CLIP score variations across CFG values, with LoRAverse remaining competitive despite a slight decrease. The bottom highlights the TCE-TIE trade-off with varying CFG values, where LoRAverse consistently achieves higher diversity, pushing the Pareto frontier toward greater semantic and visual diversity.
• Figure 5: Preference Rate of VLM-as-a-Judge. Preference rates of LoRAverse, Stylus, and SD v1.5 across visual quality, textual alignment, and diversity, as evaluated by gpt-4o. LoRAverse achieves a clear advantage in diversity while maintaining competitive visual quality and textual alignment.

Theorems & Definitions (4)

• Definition 1
• Lemma 1
• Lemma 2
• Lemma 3