OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer

TL;DR

OmniTransfer introduces a unified framework for spatio-temporal video transfer that leverages multi-view frame information and temporal cues to unify appearance and temporal transfer tasks. Its three core designs—Task-aware Positional Bias, Reference-decoupled Causal Learning, and Task-adaptive Multimodal Alignment—enable efficient and flexible control across identity, style, effects, camera movement, and motion transfer, without relying on pose priors. Empirical results show superior or competitive performance across diverse tasks, with notable improvements in appearance fidelity and temporal coherence, and a 20% reduction in runtime. This work establishes a new paradigm for high-fidelity, versatile video generation by integrating video references, diffusion modeling, and multimodal semantic guidance.

Abstract

Videos convey richer information than images or text, capturing both spatial and temporal dynamics. However, most existing video customization methods rely on reference images or task-specific temporal priors, failing to fully exploit the rich spatio-temporal information inherent in videos, thereby limiting flexibility and generalization in video generation. To address these limitations, we propose OmniTransfer, a unified framework for spatio-temporal video transfer. It leverages multi-view information across frames to enhance appearance consistency and exploits temporal cues to enable fine-grained temporal control. To unify various video transfer tasks, OmniTransfer incorporates three key designs: Task-aware Positional Bias that adaptively leverages reference video information to improve temporal alignment or appearance consistency; Reference-decoupled Causal Learning separating reference and target branches to enable precise reference transfer while improving efficiency; and Task-adaptive Multimodal Alignment using multimodal semantic guidance to dynamically distinguish and tackle different tasks. Extensive experiments show that OmniTransfer outperforms existing methods in appearance (ID and style) and temporal transfer (camera movement and video effects), while matching pose-guided methods in motion transfer without using pose, establishing a new paradigm for flexible, high-fidelity video generation.

Figures (18)

• Figure 1: OmniTransfer seamlessly unifies spatial appearance (ID and style) and temporal video transfer tasks (effect, motion and camera movement) within a single framework, and exhibits strong generalization across unseen task combinations.
• Figure 2: OmniTransfer comprises three key components: 1) Task-aware Positional Bias: exploits the model’s inherent spatial and temporal context capabilities for diverse tasks. 2) Reference-decoupled Causal Learning: separates reference and target branches for causal and efficient transfer. 3) Task-adaptive Multimodal Alignment: leverages an MLLM to unify and enhance semantic understanding across tasks.
• Figure 3: Video diffusion models are inherently capable of handling temporal consistency through spatial context.
• Figure 4: Qualitative comparison for appearance video transfer. The yellow box highlights how OmniTransfer captures richer appearance details from multiple cross-view video frames. The red box denotes the input image for image-reference methods.
• Figure 5: Qualitative comparison for appearance video transfer.