MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation

TL;DR

MoMA addresses open-vocabulary, tuning-free personalized image generation by using a Multimodal Large Language Model as a generative image-feature decoder that contextualizes a reference subject with text prompts. It introduces a self-attention feature transfer pathway with iterative masking to preserve subject identity while enabling background and texture changes. The learning framework comprises two stages: multimodal learning to produce context-aware image embeddings and diffusion learning to convert these embeddings into high-fidelity images, all while keeping the diffusion backbone frozen. Empirically, MoMA outperforms existing tuning-free and tunable baselines on both recontextualization and texture editing tasks and is released as open-source to broad adoption.

Abstract

In this paper, we present MoMA: an open-vocabulary, training-free personalized image model that boasts flexible zero-shot capabilities. As foundational text-to-image models rapidly evolve, the demand for robust image-to-image translation grows. Addressing this need, MoMA specializes in subject-driven personalized image generation. Utilizing an open-source, Multimodal Large Language Model (MLLM), we train MoMA to serve a dual role as both a feature extractor and a generator. This approach effectively synergizes reference image and text prompt information to produce valuable image features, facilitating an image diffusion model. To better leverage the generated features, we further introduce a novel self-attention shortcut method that efficiently transfers image features to an image diffusion model, improving the resemblance of the target object in generated images. Remarkably, as a tuning-free plug-and-play module, our model requires only a single reference image and outperforms existing methods in generating images with high detail fidelity, enhanced identity-preservation and prompt faithfulness. Our work is open-source, thereby providing universal access to these advancements.

Figures (21)

• Figure 1: Example images generated by our open-vocabulary personalization model. without tuning. With just one image of a subject (circled in blue), our model can generate text-aligned, identity-preserved new images of the same subject with only a single forward pass. Our model supports both re-contextualization where the same subject is located in a new environment, as shown in green, and changing the texture of the subject itself, as shown in red.
• Figure 2: Model structure. (1) On top-left, we adopt a generative multimodal image decoder to extract semantic features and modify them by the target prompt. These features are projected to text space and then injected into a pretrained frozen UNet with decoupled context cross-attentions as illustrated in light red. (2) On bottom-left, to further improve detail accuracy, we forward the clear reference image ($t=0$) to the same UNet and extract self-attention features. These fine-grained features contain detailed information about the subject and are injected into UNet through decoupled object cross-attention layers as illustrated in orange. (3) The model is trained using a two-staged training pipeline: we first train the multimodal decoder (multimodal generative learning), then jointly optimize newly added attention modules in UNet.
• Figure 3: Multimodal Generative Learning and iterative Self-Attention Masking
• Figure 4: Zero-shot new context. Visualization of our generated samples for various images and prompts. Exact subject with different context.
• Figure 5: Zero-shot new texture. Visualization of our results with new texture, material, color and style. Our model correctly balances between prompt and image fidelity.