Mod-Adapter: Tuning-Free and Versatile Multi-concept Personalization via Modulation Adapter

TL;DR

This paper tackles the problem of multi-concept personalization in text-to-image generation, including abstract concepts, without test-time fine-tuning. It introduces Mod-Adapter, a tuning-free module that leverages vision-language cross-attention and Mixture-of-Experts to predict per-block modulation directions in the DiT modulation space, enabling localized control of both object and abstract concepts. A Vision-Language Model guided pretraining strategy initializes Mod-Adapter with semantic supervision, improving training efficiency and performance. Extensive experiments on DreamBench-Abs demonstrate state-of-the-art concept preservation and prompt fidelity, as validated by quantitative metrics and human studies, with ablations confirming the importance of each component. Overall, the approach delivers versatile, scalable personalization while keeping the backbone model frozen, offering practical impact for personalized generation tasks in design, storytelling, and interactive media.

Abstract

Personalized text-to-image generation aims to synthesize images of user-provided concepts in diverse contexts. Despite recent progress in multi-concept personalization, most are limited to object concepts and struggle to customize abstract concepts (e.g., pose, lighting). Some methods have begun exploring multi-concept personalization supporting abstract concepts, but they require test-time fine-tuning for each new concept, which is time-consuming and prone to overfitting on limited training images. In this work, we propose a novel tuning-free method for multi-concept personalization that can effectively customize both object and abstract concepts without test-time fine-tuning. Our method builds upon the modulation mechanism in pre-trained Diffusion Transformers (DiTs) model, leveraging the localized and semantically meaningful properties of the modulation space. Specifically, we propose a novel module, Mod-Adapter, to predict concept-specific modulation direction for the modulation process of concept-related text tokens. It introduces vision-language cross-attention for extracting concept visual features, and Mixture-of-Experts (MoE) layers that adaptively map the concept features into the modulation space. Furthermore, to mitigate the training difficulty caused by the large gap between the concept image space and the modulation space, we introduce a VLM-guided pre-training strategy that leverages the strong image understanding capabilities of vision-language models to provide semantic supervision signals. For a comprehensive comparison, we extend a standard benchmark by incorporating abstract concepts. Our method achieves state-of-the-art performance in multi-concept personalization, supported by quantitative, qualitative, and human evaluations.

Figures (12)

• Figure 1: Results of our multi-concept personalized image generation method. Our method enables customizing both object and abstract concepts (e.g., pose, light, surface) without test-time fine-tuning. The colored words in the prompt below image indicate concepts to be personalized.
• Figure 2: Overview of the proposed method.(a) During training, the proposed Mod-Adapter module takes as input a concept image and its corresponding concept word, and predicts a concept-specific modulation direction for each DiT block. The predicted directions are integrated into the modulation (Mod) process of the concept-related text tokens in DiT. (b) Pre-training of the Mod-Adapter module. The concept image is fed into a vision-language model (VLM) to obtain a detailed descriptive caption of the target concept in the image, which is further encoded by a CLIP text encoder and mapped by an MLP layer ($\mathcal{M}$) into the DiT modulation space. The resulting feature provides the semantic supervision signals for Mod-Adapter. (c) At inference, Mod-Adapter predicts a modulation direction for each customized concept. These directions are integrated into the modulation process of their corresponding text tokens to enable multi-concept customization.
• Figure 3: Qualitative comparison. The left dashed box shows input concept images. Colored words in the prompt indicate concepts to be personalized, while underlined text highlights elements that reflect differences in prompt alignment performance between methods.
• Figure 4: Qualitative ablation results. The left dashed box shows input concept images. Eliminating any proposed component degrades qualitative performance.
• Figure 5: Screenshot of our user study rating interface. (a) Single-concept personalization evaluation. (b) Multi-concept personalization evaluation.