Owl-1: Omni World Model for Consistent Long Video Generation

TL;DR

Owl-1 tackles the problem of inconsistency in long video generation by proposing an Omni World Model that evolves a latent state $\mathbf{s}_t$ to capture long-term scene information, decodes explicit observations $\mathbf{o}_t$ with a state decoder, and forecasts future world dynamics $\mathbf{d}_t$ to update the state. The approach combines a pretrained large multimodal model (LMM) to model the state-observation-dynamics loop and a video diffusion model to generate short video clips, with a multi-stage training scheme that aligns the LMM with the diffusion model, pretrains on general videos, and then learns the long-horizon dynamics from dense captions. Key contributions include the state-observation-dynamics triplet, comprehensive latent-state conditioning, dynamics anticipation, and a practical three-stage training protocol that leverages vast short-video data for scalable long-video generation. The results show Owl-1 achieving competitive performance on VBench-I2V and VBench-Long, demonstrating improved long-horizon coherence and content diversity with potential for controllable scene transitions and world modeling in multimodal settings.

Abstract

Video generation models (VGMs) have received extensive attention recently and serve as promising candidates for general-purpose large vision models. While they can only generate short videos each time, existing methods achieve long video generation by iteratively calling the VGMs, using the last-frame output as the condition for the next-round generation. However, the last frame only contains short-term fine-grained information about the scene, resulting in inconsistency in the long horizon. To address this, we propose an Omni World modeL (Owl-1) to produce long-term coherent and comprehensive conditions for consistent long video generation. As videos are observations of the underlying evolving world, we propose to model the long-term developments in a latent space and use VGMs to film them into videos. Specifically, we represent the world with a latent state variable which can be decoded into explicit video observations. These observations serve as a basis for anticipating temporal dynamics which in turn update the state variable. The interaction between evolving dynamics and persistent state enhances the diversity and consistency of the long videos. Extensive experiments show that Owl-1 achieves comparable performance with SOTA methods on VBench-I2V and VBench-Long, validating its ability to generate high-quality video observations. Code: https://github.com/huang-yh/Owl.

Figures (6)

• Figure 1: Owl-1 approaches consistent long video generation with an omni world model, which models the evolution of the underlying world with latent state, explicit observation and world dynamics variables.
• Figure 2: Iterative long video generation. Conventional iterative long video generation methods use the last-frame output as the condition for the next-round generation, which lacks long-term consistency. Our method constructs an omni world model for comprehensive conditioning.
• Figure 3: Overall framework. Our Owl-1 models the evolution of the world with the latent state variables $\mathbf{s}$, and film them into video observations $\mathbf{o}$ along the generation process. We also incorporate anticipation of the world dynamics $\mathbf{d}$ to explicitly drive the evolution.
• Figure 4: Video frames visualization results for general video generation. We sample 5 frames from each of our generated videos, which lasts 8 seconds. Our Owl-1 generates videos covering various topics with good quality.
• Figure 5: Video frames visualization results for world model based video generation. We generate 3 scenes for each prompt, and sample 2 frames from each scene. Every scene lasts for 8 seconds, and the whole video is 24 seconds long. Our Owl-1 generates consistent long videos with reasonable dynamics anticipation. Blue and red texts denote given prompt and predicted dynamics, respectively.