VINO: A Unified Visual Generator with Interleaved OmniModal Context

TL;DR

VINO addresses fragmentation in diffusion-based visual generation by unifying image and video generation and editing under a single diffusion backbone conditioned on interleaved omni-modal context. It couples a vision–language model (VLM) with a Multimodal Diffusion Transformer (MMDiT), uses learnable query tokens, and introduces a token-boundary mechanism to preserve identity across references by grounding semantic and latent representations. A progressive three-stage training pipeline aligns long-form captions with short editing prompts, enabling robust multitask capabilities while retaining priors from the base video model. Empirically, VINO delivers strong visual quality, faithful instruction following, and improved identity preservation across both images and videos, demonstrating a scalable path toward general-purpose multimodal generation and editing.

Abstract

We present VINO, a unified visual generator that performs image and video generation and editing within a single framework. Instead of relying on task-specific models or independent modules for each modality, VINO uses a shared diffusion backbone that conditions on text, images and videos, enabling a broad range of visual creation and editing tasks under one model. Specifically, VINO couples a vision-language model (VLM) with a Multimodal Diffusion Transformer (MMDiT), where multimodal inputs are encoded as interleaved conditioning tokens, and then used to guide the diffusion process. This design supports multi-reference grounding, long-form instruction following, and coherent identity preservation across static and dynamic content, while avoiding modality-specific architectural components. To train such a unified system, we introduce a multi-stage training pipeline that progressively expands a video generation base model into a unified, multi-task generator capable of both image and video input and output. Across diverse generation and editing benchmarks, VINO demonstrates strong visual quality, faithful instruction following, improved reference and attribute preservation, and more controllable multi-identity edits. Our results highlight a practical path toward scalable unified visual generation, and the promise of interleaved, in-context computation as a foundation for general-purpose visual creation.

Figures (13)

• Figure 1: Showcase of VINO in image generation and image editing.
• Figure 2: Showcase of VINO in video generation and video editing.
• Figure 3: Overview of the VINO pipeline. Our unified framework conditions generation on an interleaved omnimodal context that jointly encodes system prompts, prompts/instructions, reference images/videos, and learnable tokens. A frozen VLM processes textual instructions together with visual references, producing multimodal embeddings that are augmented with learnable tokens (purple) and separated by special tokens (vision start token and vision end token ). These interleaved multimodal representations are fed into the MMDiT blocks, which also receive VAE latents from the reference images or video. The MMDiT model performs denoising conditioned on the full multimodal context, enabling VINO to execute image and video generation as well as instruction-based editing within a single unified architecture.
• Figure 4: System prompt for different condition task, where <|user_text|>, <|user_image|> and <|user_video|> denote the user-provided input conditions across different modalities. For brevity, we omit the <|vision_start|> and <|vision_end|> tokens for the visual modalities.
• Figure 5: 3D RoPE strategy for the VAE branch in VINO. We apply a unified 3D RoPE schedule along the temporal axis to interleave different visual modalities in the MMDiT VAE branch. Each modality—single reference images, multi-frame reference videos, and noisy target latents—is placed on a shared RoPE timeline, separated by special tokens, which are projected from the VLM output. This structured RoPE layout enables the model to distinguish heterogeneous visual sources.