DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in Multimodal Cycles

TL;DR

DoraCycle introduces a domain-oriented adaptation framework that leverages two multimodal cycles on a single unified transformer to adapt to target domains with unpaired data. By enforcing cycle-consistency at endpoints and stabilizing training with EMA targets and gradient control, it enables effective stylization and domain alignment, and extends to identity-specific adaptations when small amounts of paired data are available. The approach demonstrates competitive performance against DreamBooth and ITIT across data regimes while significantly reducing the need for paired text–image data. The method broadens practical applications of unified multimodal models by enabling scalable, data-efficient domain adaptation.

Abstract

Adapting generative models to specific domains presents an effective solution for satisfying specialized requirements. However, adapting to some complex domains remains challenging, especially when these domains require substantial paired data to capture the targeted distributions. Since unpaired data from a single modality, such as vision or language, is more readily available, we utilize the bidirectional mappings between vision and language learned by the unified generative model to enable training on unpaired data for domain adaptation. Specifically, we propose DoraCycle, which integrates two multimodal cycles: text-to-image-to-text and image-to-text-to-image. The model is optimized through cross-entropy loss computed at the cycle endpoints, where both endpoints share the same modality. This facilitates self-evolution of the model without reliance on annotated text-image pairs. Experimental results demonstrate that for tasks independent of paired knowledge, such as stylization, DoraCycle can effectively adapt the unified model using only unpaired data. For tasks involving new paired knowledge, such as specific identities, a combination of a small set of paired image-text examples and larger-scale unpaired data is sufficient for effective domain-oriented adaptation. The code will be released at https://github.com/showlab/DoraCycle.

Figures (8)

• Figure 1: Training paradigms for unified generative models. (a) Traditional training involves using paired image-text data and optimizing the unified model with paired losses for both image-to-text (I2T) and text-to-image (T2I) generation tasks. (b) In contrast, the proposed multimodal cycle training framework leverages unpaired images and texts. By using cycle-consistency losses, the unified model learns to maintain consistency between input and output across modalities, enabling adaptation without the need for extensive paired datasets.
• Figure 2: The overview of T cycle (text-to-image-to-text) of the proposed DoraCycle. The I cycle is similar and is omitted in the figure for brevity.
• Figure 3: Domain-oriented adaptation with different training setups. (a) Image generated by the base model without training for adoption. (b) Image generated by the model trained with 10 paired image-text samples. (c) Image generated by the model trained with 300 paired image-text samples. (d) Image generated by the model trained by DoraCycle on only unpaired data. (e) Image-to-Text-to-Image translation performed by the adapted model trained by DoraCycle.
• Figure 4: Image-to-text and text-to-image generation by the unified models that adapted for two domains. The special tokens are omitted.
• Figure 5: Effect of special tokens on character learning. (a) Base model without training. (b) Model trained without using special tokens, showing attribute confusion among characters. (c) Model trained with special tokens, improving character attribute alignment and reducing confusion.