DCoAR: Deep Concept Injection into Unified Autoregressive Models for Personalized Text-to-Image Generation

TL;DR

DCoAR tackles personalized text-to-image generation in unified autoregressive models by introducing deep concept injection through Layer-wise Multimodal Context Learning, while keeping the backbone frozen. It couples LMCL with Dual Prior Preservation and Context-Aware Self-Regularization to maintain subject fidelity and re-contextualization, and enables training-free subject–style composition by token concatenation with an Identity Mask. Empirical results on DreamBench and StyleDrop demonstrate state-of-the-art personalization with minimal trainable parameters and competitive styling without additional training, validated by extensive ablations. The work highlights the importance of injecting concepts across layers and balancing priors and contextual flexibility for scalable, high-fidelity personalization in multimodal AR systems.

Abstract

The unified autoregressive (AR) model excels at multimodal understanding and generation. However, its full potential in the domain of customized image generation has yet to be fully realized. Existing customization approaches for unified AR models face a fundamental dilemma: adaptation-based methods suffer from overfitting and scalability bottlenecks, while concept-injection paradigms are constrained by a shallow injection strategy that leads to poor visual fidelity and impaired re-contextualization. To address this, we propose DCoAR, a novel deep concept injection framework that maintains a completely frozen pre-trained model. DCoAR deeply integrates new concepts through a Layer-wise Multimodal Context Learning (LMCL) strategy, which is stabilized by a multi-faceted regularization scheme: a Dual Prior Preservation (DPP) loss to mitigate semantic drift and a Context-Aware Self-Regularization (CASR) loss to enhance re-contextualization. The framework also enables training-free subject customization in user-provided styles. Experiments demonstrate that DCoAR significantly outperforms previous injection-based methods and achieves performance competitive with adaptation-based approaches while requiring substantially fewer trainable parameters. Code: https://github.com/KZF-kzf/CoAR

Figures (16)

• Figure 1: Visualization of samples generated by our proposed DCoAR, showcasing subject personalization (left), where a few reference images are provided, and style personalization (right), where the subject is rendered in user-specified artistic styles without additional training.
• Figure 2: Overview of the proposed DCoAR framework for subject-driven personalization in multi-modal autoregressive models. (a) Layerwise Multimodal Context Learning, where learnable context tokens are injected into multiple Transformer layers for concept representation. (b) Dual Prior Preservation (DPP) regularizes the customized distribution against the pre-trained model to mitigate overfitting and language drift. (c) Context-Aware Self-Regularization (CASR) initializes and constrains context tokens towards the subject embedding space to enhance fidelity and re-contextualization. (d) Training-free subject–style composition by directly combining subject and style tokens to enable flexible customized generation.
• Figure 3: Qualitative comparison of subject-driven personalization on the DreamBench benchmark. As concept-injection methods generally exhibit lower visual fidelity, we focus the qualitative comparison on the more competitive adaptation-based methods. Additional comparisons are available in the supplementary material.
• Figure 4: Training-free subject-style generation with our DCoAR on DreamBench ruiz2023dreambooth and StyleDrop sohn2023styledrop datasets.
• Figure 5: Re-contextualization with our DCoAR on DreamBench ruiz2023dreambooth and StyleDrop sohn2023styledrop datasets.