TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation

TL;DR

TI2V seeks to generate controllable videos from a single frame and a text description, but is hindered by misalignment between text-guided motion and object identities as well as lower perceptual quality. The authors introduce TIV-Diffusion, a diffusion-based TI2V framework that uses object-centric object slots via Slot Attention and an Object Disentanglement Fusion Module to align textual descriptions with per-object motion, modulated through SPADE and integrated autoregressively with ConvGRU temporal conditioning. A key contribution is adaptive slot-text alignment and slot-conditioned conditioning in the diffusion process, which mitigates object disappearance and deformation while improving semantic consistency. Extensive experiments on MNIST and CATER, plus additional real-world datasets, show state-of-the-art performance in FID, FVD, and LPIPS, demonstrating stronger cross-modal alignment and temporal coherence for text-driven video generation. The approach offers a robust pathway to high-quality, controllable TI2V with notable practical impact for video creation guided by natural language.

Abstract

Text-driven Image to Video Generation (TI2V) aims to generate controllable video given the first frame and corresponding textual description. The primary challenges of this task lie in two parts: (i) how to identify the target objects and ensure the consistency between the movement trajectory and the textual description. (ii) how to improve the subjective quality of generated videos. To tackle the above challenges, we propose a new diffusion-based TI2V framework, termed TIV-Diffusion, via object-centric textual-visual alignment, intending to achieve precise control and high-quality video generation based on textual-described motion for different objects. Concretely, we enable our TIV-Diffuion model to perceive the textual-described objects and their motion trajectory by incorporating the fused textual and visual knowledge through scale-offset modulation. Moreover, to mitigate the problems of object disappearance and misaligned objects and motion, we introduce an object-centric textual-visual alignment module, which reduces the risk of misaligned objects/motion by decoupling the objects in the reference image and aligning textual features with each object individually. Based on the above innovations, our TIV-Diffusion achieves state-of-the-art high-quality video generation compared with existing TI2V methods.

Figures (29)

• Figure 1: Humans naturally disentangle different objects in their environment. In the image above, the human wants to 'Put the oranges on the table.' First, he will decouple objects with different attributes, then find the target objects according to the intention and complete the action. Our model draws inspiration from this observation.
• Figure 2: Illustration of the TIV-Diffusion framework. Given an image and its corresponding text caption, TIV-Diffusion can autoregressively generate subsequent video frames. To improve the alignment between objects' appearance and movement from different modalities, we introduce object-centric representations. We employ a slot attention encoder to extract slots and further fuse them with text information through the Object Distanglement Fusion Module (ODFM), which will be introduced thoroughly in Sec. \ref{['sec:Utilize Object Disentanglement for Improved Alignment']}. Additionally, the video generation process adaptively incorporates object attribute information from slots to address object deformation and disappearance.
• Figure 3: Object Distanglement Fusion Module (ODFM). We initially identify the relevant target moving objects within the text caption based on the provided slots and then align slots with the text caption individually. Subsequently, we utilize the text-enhanced slots to facilitate the video generation process.
• Figure 4: Comparison Results on Double Moving MNIST. Under the constraints of object-centric representations, the shapes of the digits in the videos generated by our model remain consistent even when they overlap. To facilitate visualization, we have extracted specific frames.
• Figure 5: Comparison Results on CATER-GEN-v2. We propose object disentanglement for enhanced alignment between text and image in video generation. This yields improved semantic consistency, enabling precise identification and motion in accordance with textual descriptions.