SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data

TL;DR

SELMA tackles the challenge of faithful text-to-image generation by automatically generating multi-skill image-text data with an LLM, training skill-specific LoRA experts, and merging them into a single multi-skill T2I model. The approach reduces reliance on human-annotated data and mitigates knowledge conflicts through inference-time LoRA merging, achieving state-of-the-art text faithfulness on DSG and TIFA across SD v1.4, v2, and SDXL. It also shows that auto-generated data can match or exceed ground-truth data in effectiveness and reveals a weak-to-strong generalization where weaker-image generators can benefit stronger models. Overall, SELMA offers a scalable pathway to build faithful, flexible T2I systems capable of handling diverse prompts without expensive human supervision.

Abstract

Recent text-to-image (T2I) generation models have demonstrated impressive capabilities in creating images from text descriptions. However, these T2I generation models often fall short of generating images that precisely match the details of the text inputs, such as incorrect spatial relationship or missing objects. In this paper, we introduce SELMA: Skill-Specific Expert Learning and Merging with Auto-Generated Data, a novel paradigm to improve the faithfulness of T2I models by fine-tuning models on automatically generated, multi-skill image-text datasets, with skill-specific expert learning and merging. First, SELMA leverages an LLM's in-context learning capability to generate multiple datasets of text prompts that can teach different skills, and then generates the images with a T2I model based on the prompts. Next, SELMA adapts the T2I model to the new skills by learning multiple single-skill LoRA (low-rank adaptation) experts followed by expert merging. Our independent expert fine-tuning specializes multiple models for different skills, and expert merging helps build a joint multi-skill T2I model that can generate faithful images given diverse text prompts, while mitigating the knowledge conflict from different datasets. We empirically demonstrate that SELMA significantly improves the semantic alignment and text faithfulness of state-of-the-art T2I diffusion models on multiple benchmarks (+2.1% on TIFA and +6.9% on DSG), human preference metrics (PickScore, ImageReward, and HPS), as well as human evaluation. Moreover, fine-tuning with image-text pairs auto-collected via SELMA shows comparable performance to fine-tuning with ground truth data. Lastly, we show that fine-tuning with images from a weaker T2I model can help improve the generation quality of a stronger T2I model, suggesting promising weak-to-strong generalization in T2I models.

Figures (8)

• Figure 1: Comparison of different fine-tuning paradigms for text-to-image (T2I) generation models. (a) Supervised Fine-tuning (SFT): a T2I model is trained with image-text pairs from existing datasets. (b) Fine-tuning with Human Preference (e.g., RL/DPO): humans annotate their preferences on images by ranking/scoring in terms of text alignments, and a T2I model is trained to maximize the human preference scores. (c) SELMA: instead of collecting image-text pairs or human preference annotations, we automatically collect image-text pairs for desired skills with LLM and T2I model, and create a multi-skill T2I model by learning and merging skill-specific expert models.
• Figure 2: Illustration of the four-stage pipeline of SELMA. (a) Prompt Generation: Given a short skill description and a few (i.e., three) seed examples about a specific skill, we generate prompts to teach the skill with an LLM, while maintaining prompt diversity via text-similarity based filtering. (b) Image Generation: Given the LLM-generated text prompts, we generate training images with a T2I model. (c) Skill-Specific Expert Learning: We learn skill-specific expert T2I models based on LoRA fine-tuning. (d) Merging Expert Models: We obtain a multi-skill T2I model by merging the skill-specific LoRA parameters.
• Figure 3: DSG accuracy of SD v2 fine-tuned with different image-text pairs.
• Figure 4: Human Evaluation on 200 sampled text prompts from DSG, where we show the win vs. lose percentages of SDXL and SDXL+SELMA (Ours).
• Figure 5: Example images generated with SDXL and SDXL+SELMA. SELMA shows better performance in object composition, attribute binding, and long text prompt following. We highlight the parts of the prompts in red where SDXL makes errors while SDXL+SELMA generates correctly.