DuoGen: Towards General Purpose Interleaved Multimodal Generation

TL;DR

DuoGen tackles the challenge of general-purpose interleaved multimodal generation by decoupling text understanding/generation from image synthesis, leveraging a pretrained multimodal LLM (MLLM) and a diffusion transformer (DiT) pretrained on video data. A two-stage training strategy first tunes the MLLM on high-quality interleaved data, then aligns the DiT with the MLLM using interleaved image–text sequences, enabling robust text- and image-wise reasoning, drafting, and editing. The authors curate a 298k instruction-tuning dataset plus large-scale interleaved alignment data, and introduce a dedicated interleaved benchmark, showing DuoGen outperforms open-source baselines and approaches top commercial systems across text quality, image fidelity, and image-context alignment, with strong performance in text-to-image and editing tasks. The work provides substantial data, architectural design, and evaluation resources to advance general-purpose interleaved multimodal generation and lays groundwork for scalable cross-modal reasoning and generation.

Abstract

Interleaved multimodal generation enables capabilities beyond unimodal generation models, such as step-by-step instructional guides, visual planning, and generating visual drafts for reasoning. However, the quality of existing interleaved generation models under general instructions remains limited by insufficient training data and base model capacity. We present DuoGen, a general-purpose interleaved generation framework that systematically addresses data curation, architecture design, and evaluation. On the data side, we build a large-scale, high-quality instruction-tuning dataset by combining multimodal conversations rewritten from curated raw websites, and diverse synthetic examples covering everyday scenarios. Architecturally, DuoGen leverages the strong visual understanding of a pretrained multimodal LLM and the visual generation capabilities of a diffusion transformer (DiT) pretrained on video generation, avoiding costly unimodal pretraining and enabling flexible base model selection. A two-stage decoupled strategy first instruction-tunes the MLLM, then aligns DiT with it using curated interleaved image-text sequences. Across public and newly proposed benchmarks, DuoGen outperforms prior open-source models in text quality, image fidelity, and image-context alignment, and also achieves state-of-the-art performance on text-to-image and image editing among unified generation models. Data and code will be released at https://research.nvidia.com/labs/dir/duetgen/.

Figures (24)

• Figure 1: Capabilities of DuoGen. Beyond standard tasks like image understanding, generation, editing, and navigation, DuoGen supports interleaved multimodal content generation, like step-by-step tutorials or cooking recipe.
• Figure 2: Data engine for processing website data. We design a data engine consists of a series of filtering and rewriting steps to convert noisy website data into high-quality instruction tuning data for interleaved generation.
• Figure 3: Architecture and training strategy of DuoGen. DuoGen consists of a pretrained multimodal large language model (MLLM) and diffusion transformer (DiT) pretrained on video generation. If a "< Begin-of-Vision>" (BOV) token is generated by the MLLM, then all the images in the interleaved sequence is packed as "condition frames" to the DiT and the MLLM hidden-states before the < BOV> token is sent to the DiT as the text condition to generate the new images.
• Figure 4: Generation results of DuoGen on Cooking-200.
• Figure 5: Generation results of DuoGen on Cooking-200.