EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM

TL;DR

EasyRef introduces a plug-and-play framework that conditions diffusion models on multiple reference images and text prompts by leveraging a generalist multimodal LLM for multi-image understanding. It encodes references into a compact set of learnable tokens, projects them into the diffusion process via adapters, and employs a progressive training scheme across alignment pretraining, single-reference finetuning, and multi-reference finetuning. The approach demonstrates superior multi-reference generation quality and zero-shot generalization on MRBench and COCO, outperforming IP-Adapter and LoRA baselines while remaining compatible with ControlNet. The work also provides MRBench as a high-signal benchmark for evaluating multi-reference image generation. Overall, EasyRef advances robust, scalable multi-reference conditioning for diffusion models using a single generalist MLLM.

Abstract

Significant achievements in personalization of diffusion models have been witnessed. Conventional tuning-free methods mostly encode multiple reference images by averaging their image embeddings as the injection condition, but such an image-independent operation cannot perform interaction among images to capture consistent visual elements within multiple references. Although the tuning-based Low-Rank Adaptation (LoRA) can effectively extract consistent elements within multiple images through the training process, it necessitates specific finetuning for each distinct image group. This paper introduces EasyRef, a novel plug-and-play adaptation method that enables diffusion models to be conditioned on multiple reference images and the text prompt. To effectively exploit consistent visual elements within multiple images, we leverage the multi-image comprehension and instruction-following capabilities of the multimodal large language model (MLLM), prompting it to capture consistent visual elements based on the instruction. Besides, injecting the MLLM's representations into the diffusion process through adapters can easily generalize to unseen domains, mining the consistent visual elements within unseen data. To mitigate computational costs and enhance fine-grained detail preservation, we introduce an efficient reference aggregation strategy and a progressive training scheme. Finally, we introduce MRBench, a new multi-reference image generation benchmark. Experimental results demonstrate EasyRef surpasses both tuning-free methods like IP-Adapter and tuning-based methods like LoRA, achieving superior aesthetic quality and robust zero-shot generalization across diverse domains.

Figures (12)

• Figure 1: Spatial misalignment issue of the embedding averaging operation. The images with faces are synthetic.
• Figure 2: Overview of EasyRef with SDXL. EasyRef extracts consistent visual eliments from multiple reference images and the text prompt via a MLLM, injecting the condition representations into the diffusion model through cross-attention layers. We only plot 1 cross-attention layer for simplicity.
• Figure 2: More generated samples of character consistency with EasyRef in a zero-shot setting.
• Figure 3: Distribution of our curated dataset.
• Figure 3: More generated samples of style consistency with EasyRef in a zero-shot setting.