MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Experts

TL;DR

A simple yet effective method called Mixture of Low-rank Experts (MoLE) by considering low-rank modules trained on close-up hand and face images respectively as experts by considering low-rank modules trained on close-up hand and face images respectively as experts is proposed.

Abstract

Text-to-image diffusion has attracted vast attention due to its impressive image-generation capabilities. However, when it comes to human-centric text-to-image generation, particularly in the context of faces and hands, the results often fall short of naturalness due to insufficient training priors. We alleviate the issue in this work from two perspectives. 1) From the data aspect, we carefully collect a human-centric dataset comprising over one million high-quality human-in-the-scene images and two specific sets of close-up images of faces and hands. These datasets collectively provide a rich prior knowledge base to enhance the human-centric image generation capabilities of the diffusion model. 2) On the methodological front, we propose a simple yet effective method called Mixture of Low-rank Experts (MoLE) by considering low-rank modules trained on close-up hand and face images respectively as experts. This concept draws inspiration from our observation of low-rank refinement, where a low-rank module trained by a customized close-up dataset has the potential to enhance the corresponding image part when applied at an appropriate scale. To validate the superiority of MoLE in the context of human-centric image generation compared to state-of-the-art, we construct two benchmarks and perform evaluations with diverse metrics and human studies. Datasets, model, and code are released at https://sites.google.com/view/mole4diffuser/.

Figures (24)

• Figure 1: Compare MoLE with other diffusion models. Pay more attention to the face and (especially) hand. Zoom in for a better view.
• Figure 2: Inspiration of Mixture of Low-rank Experts. In the first and second row, we train two low-rank modules on SD v1.5 simply using off-the-shelf Celeb-HQ face dataset karras2018progressive and 11k Hands dataset afifi201911k, respectively. With a proper scale weight, low-rank module can refine corresponding part. We term this phenomenon as low-rank refinement.
• Figure 3: Some showcases of our human-centric dataset.
• Figure 4: The results of four captioning models. Texts in red are inaccurate descriptions and texts in green are detailed correct descriptions. LLaVA presents a good balance between the level of detail and error rate, and thus is chosen for captioning our dataset.
• Figure 5: The framework of MoLE. $X$ is the input of any linear layers in UNet. $A$ and $B$ are low-rank matrices.