Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

One-Shot Learning Meets Depth Diffusion in Multi-Object Videos

Anisha Jain

TL;DR

The paper tackles the challenge of generating coherent, multi-object videos from minimal data by leveraging a depth-conditioned latent diffusion framework. It extends a pre-trained depth-conditioned Text-to-Image Diffusion Model with pseudo-3D convolutions, temporal self-attention, and a sparse update scheme that only tunes projection matrices, while DDIM inversion provides structure guidance for inference. The proposed approach yields temporally coherent videos across diverse artistic styles by explicitly modeling depth and object interactions in the latent space. Empirical results show improved frame consistency and textual faithfulness over baselines, demonstrating the viability of one-shot training for complex multi-object video synthesis with meaningful depth control.

Abstract

Creating editable videos that depict complex interactions between multiple objects in various artistic styles has long been a challenging task in filmmaking. Progress is often hampered by the scarcity of data sets that contain paired text descriptions and corresponding videos that showcase these interactions. This paper introduces a novel depth-conditioning approach that significantly advances this field by enabling the generation of coherent and diverse videos from just a single text-video pair using a pre-trained depth-aware Text-to-Image (T2I) model. Our method fine-tunes the pre-trained model to capture continuous motion by employing custom-designed spatial and temporal attention mechanisms. During inference, we use the DDIM inversion to provide structural guidance for video generation. This innovative technique allows for continuously controllable depth in videos, facilitating the generation of multiobject interactions while maintaining the concept generation and compositional strengths of the original T2I model across various artistic styles, such as photorealism, animation, and impressionism.

One-Shot Learning Meets Depth Diffusion in Multi-Object Videos

TL;DR

The paper tackles the challenge of generating coherent, multi-object videos from minimal data by leveraging a depth-conditioned latent diffusion framework. It extends a pre-trained depth-conditioned Text-to-Image Diffusion Model with pseudo-3D convolutions, temporal self-attention, and a sparse update scheme that only tunes projection matrices, while DDIM inversion provides structure guidance for inference. The proposed approach yields temporally coherent videos across diverse artistic styles by explicitly modeling depth and object interactions in the latent space. Empirical results show improved frame consistency and textual faithfulness over baselines, demonstrating the viability of one-shot training for complex multi-object video synthesis with meaningful depth control.

Abstract

Creating editable videos that depict complex interactions between multiple objects in various artistic styles has long been a challenging task in filmmaking. Progress is often hampered by the scarcity of data sets that contain paired text descriptions and corresponding videos that showcase these interactions. This paper introduces a novel depth-conditioning approach that significantly advances this field by enabling the generation of coherent and diverse videos from just a single text-video pair using a pre-trained depth-aware Text-to-Image (T2I) model. Our method fine-tunes the pre-trained model to capture continuous motion by employing custom-designed spatial and temporal attention mechanisms. During inference, we use the DDIM inversion to provide structural guidance for video generation. This innovative technique allows for continuously controllable depth in videos, facilitating the generation of multiobject interactions while maintaining the concept generation and compositional strengths of the original T2I model across various artistic styles, such as photorealism, animation, and impressionism.
Paper Structure (13 sections, 8 equations, 5 figures, 1 table)

This paper contains 13 sections, 8 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-Image Synthesis
  4. Text-to-Video Synthesis
  5. Depth-guided Image and Video Synthesis
  6. Methodology
  7. Premilinary: Latent Diffusion Model
  8. Depth-guided Text-to-Video Generation
  9. Experiments
  10. Implementation Details
  11. Qualitative Results
  12. Quantitative Comparison
  13. Conclusion

Figures (5)

  • Figure 1: Using a text-video pair (for example, “a polar bear with her cubs”) as input, our approach utilizes pretrained depth-conditioned T2I diffusion models to generate T2V content. During fine-tuning, we update the projection matrices in attention blocks using the standard diffusion training loss and regularization.
  • Figure 2: During inference, we generate a new video by sampling from the latent noise inverted from the input video, using an edited prompt (e.g., “A tigress with her cubs in a forest”) as guidance.
  • Figure 3: Sample results of our method. The first row consists of images from the input sequence along with the text prompt. The rows below consist of images from generated sequences for the edited prompts for different styles and novelty.
  • Figure 4: On the left, Cog-Video produces undesirable results for human interactions with blurred faces and weird hands. While on the right, Tune-A-Video produces unpleasant results when the input video contains multiple objects and exhibits occlusions. For example, the two pandas at the bottom being mixed together.
  • Figure 5: Some more results of our method. The first and third row show frames from the input sequence. The second and fourth row illustrate the frames from the respective generated sequences for the edited prompt.