MUSAR: Exploring Multi-Subject Customization from Single-Subject Dataset via Attention Routing

TL;DR

MUSAR tackles the data bottleneck and entanglement challenges in multi-subject text-to-image generation by learning from single-subject data. It introduces de-biased diptych learning (diptych data construction with static attention routing and dual-branch LoRA) to preserve multi-subject features while mitigating bias, and dynamic attention routing to enforce bijective region-to-subject mappings and prevent cross-subject interference. Experiments show MUSAR surpasses methods trained on large multi-subject datasets in image fidelity and subject consistency, while maintaining natural interaction across subjects. This approach enables tuning-free, scalable multi-subject customization with broad practical impact for personalized content, design, and creative AI applications.

Abstract

Current multi-subject customization approaches encounter two critical challenges: the difficulty in acquiring diverse multi-subject training data, and attribute entanglement across different subjects. To bridge these gaps, we propose MUSAR - a simple yet effective framework to achieve robust multi-subject customization while requiring only single-subject training data. Firstly, to break the data limitation, we introduce debiased diptych learning. It constructs diptych training pairs from single-subject images to facilitate multi-subject learning, while actively correcting the distribution bias introduced by diptych construction via static attention routing and dual-branch LoRA. Secondly, to eliminate cross-subject entanglement, we introduce dynamic attention routing mechanism, which adaptively establishes bijective mappings between generated images and conditional subjects. This design not only achieves decoupling of multi-subject representations but also maintains scalable generalization performance with increasing reference subjects. Comprehensive experiments demonstrate that our MUSAR outperforms existing methods - even those trained on multi-subject dataset - in image quality, subject consistency, and interaction naturalness, despite requiring only single-subject dataset.

Figures (8)

• Figure 1: Breaking the data barrier, MUSAR enables remarkable multi-subject customization from solely single-subject dataset, demonstrating scalable generalization as the number of subjects grows.
• Figure 2: A sample of diptych learning. We pairing existing single-subject data, creating multi-condition and prompts inputs, and diptych targets for effective multi-subject learning.
• Figure 3: Two strategies to mitigate learning diptych biases. (a) Static Attention Routing: a routing mechanism that prevents prompt-condition contamination and inter-condition interactions. (b) Dual-branch LoRA: specific LoRA pathways are selectively activated based on input condition types.
• Figure 4: Dynamic Attention Routing enforces a bijective mapping between noise tokens and condition subjects, effectively mitigating multi-subject feature entanglement.
• Figure 5: Visualization of the noise-condition affinity score $S^{*}$ in Dynamic Attention Routing. Each row displays dynamic routing probabilities per condition, demonstrating how adaptive attention selectively focuses on different conditions throughout the denoising process.