Tele-Omni: a Unified Multimodal Framework for Video Generation and Editing

TL;DR

Tele-Omni tackles the need for a scalable, unified framework capable of handling multimodal instructions for video generation and editing. It introduces a two-module system in which a pretrained multimodal LLM parses text, images, and reference videos to produce structured conditioning signals that guide a diffusion transformer video generator via a trainable adaptor; visual inputs are further encoded with a VAE to preserve rich spatial-temporal information. A task-aware data processing pipeline unifies heterogeneous inputs into a structured instruction format, enabling joint training across text-to-video, image-to-video, first-last-frame, in-context generation, and in-context editing. Experimental results demonstrate competitive performance across multiple tasks with strong temporal coherence and high editing fidelity, validating Tele-Omni as a flexible, scalable foundation for multimodal video synthesis and manipulation.

Abstract

Recent advances in diffusion-based video generation have substantially improved visual fidelity and temporal coherence. However, most existing approaches remain task-specific and rely primarily on textual instructions, limiting their ability to handle multimodal inputs, contextual references, and diverse video generation and editing scenarios within a unified framework. Moreover, many video editing methods depend on carefully engineered pipelines tailored to individual operations, which hinders scalability and composability. In this paper, we propose Tele-Omni, a unified multimodal framework for video generation and editing that follows multimodal instructions, including text, images, and reference videos, within a single model. Tele-Omni leverages pretrained multimodal large language models to parse heterogeneous instructions and infer structured generation or editing intents, while diffusion-based generators perform high-quality video synthesis conditioned on these structured signals. To enable joint training across heterogeneous video tasks, we introduce a task-aware data processing pipeline that unifies multimodal inputs into a structured instruction format while preserving task-specific constraints. Tele-Omni supports a wide range of video-centric tasks, including text-to-video generation, image-to-video generation, first-last-frame video generation, in-context video generation, and in-context video editing. By decoupling instruction parsing from video synthesis and combining it with task-aware data design, Tele-Omni achieves flexible multimodal control while maintaining strong temporal coherence and visual consistency. Experimental results demonstrate that Tele-Omni achieves competitive performance across multiple tasks.

Figures (8)

• Figure 1: Overview of Tele-Omni. Tele-Omni adopts a two-module architecture, where an MLLM serves as the understanding module and generates editing guidance from the input instruction and visual inputs, and a DiT acts as the generation module connected to the MLLM via an adaptor. During training, the DiT and adaptor are trainable, while the MLLM and VAE are frozen.
• Figure 2: Samples of constructed image-guided video editing dataset
• Figure 3: Example generated videos for image and text prompted video generation task.
• Figure 4: Example generated videos for first and last frame conditioned video generation task.
• Figure 5: Example generated videos for video editing task. This task is further divided into four categories: (1) Entity Editing, (2) Style Editing, (3) Remove and (4) Insertion.