WeGen: A Unified Model for Interactive Multimodal Generation as We Chat

TL;DR

WeGen tackles the challenge of building a user-friendly, interactive multimodal generation system that can adhere to references while remaining creative when prompts are vague. It fuses a Multimodal Large Language Model with a diffusion-based generator, augmented by two innovations: the Dynamic Instance Identity Consistency (DIIC) data pipeline, which learns to preserve identity across video-derived instances, and the Prompt Self-Rewriting (PSR) mechanism, which injects diversity through calibrated prompt rewriting. The approach achieves state-of-the-art results on multiple generation benchmarks, particularly in maintaining identity consistency and enabling diverse outputs, while functioning as a practical design copilot for end users. The work advances the practicality of unified multimodal generation by enabling iterative refinement, cross-task applicability, and more controllable outputs with potential for broad real-world impact.

Abstract

Existing multimodal generative models fall short as qualified design copilots, as they often struggle to generate imaginative outputs once instructions are less detailed or lack the ability to maintain consistency with the provided references. In this work, we introduce WeGen, a model that unifies multimodal generation and understanding, and promotes their interplay in iterative generation. It can generate diverse results with high creativity for less detailed instructions. And it can progressively refine prior generation results or integrating specific contents from references following the instructions in its chat with users. During this process, it is capable of preserving consistency in the parts that the user is already satisfied with. To this end, we curate a large-scale dataset, extracted from Internet videos, containing rich object dynamics and auto-labeled dynamics descriptions by advanced foundation models to date. These two information are interleaved into a single sequence to enable WeGen to learn consistency-aware generation where the specified dynamics are generated while the consistency of unspecified content is preserved aligned with instructions. Besides, we introduce a prompt self-rewriting mechanism to enhance generation diversity. Extensive experiments demonstrate the effectiveness of unifying multimodal understanding and generation in WeGen and show it achieves state-of-the-art performance across various visual generation benchmarks. These also demonstrate the potential of WeGen as a user-friendly design copilot as desired. The code and models will be available at https://github.com/hzphzp/WeGen.

Figures (11)

• Figure 1: Interactive dialogue examples between users and WeGen, demonstrating unified capabilities across diverse visual generation tasks through natural conversations.
• Figure 2: Dynamic Instance Identity Consistency (DIIC) Data-pipeline.
• Figure 3: Case studies showcasing WeGen's capabilities across various tasks, including text-to-image generation, subject-driven visual generation (both single and multiple subjects), image editing, condition-based generation (canny, depth, pose), style transfer, super-resolution, inpainting, outpainting
• Figure 4: Comparison of instance identity consistency with state-of-the-art methods.
• Figure 5: Diversity comparison of generated images with different random seeds. For each prompt, we show multiple generations from Emu-2 (left) and our method (right). Our method produces more diverse outputs while maintaining semantic consistency with the input prompts.