MVOC: a training-free multiple video object composition method with diffusion models

TL;DR

This work introduces MVOC, a training-free method for composing multiple video objects with diffusion models while preserving each object's motion and identity and enabling inter-object interactions. It consists of a preprocessing stage using DDIM inversion and object masks, followed by an image-guided first-frame edit and a video generation stage that injects per-object features and attention maps. A key contribution is the training-free multiple dependence generation, which layers object-conditioned signals to produce coherent multi-object dynamics during sampling. Extensive qualitative and quantitative experiments show MVOC achieves superior temporal consistency and interaction-aware harmonization compared with state-of-the-art baselines, albeit with some hyperparameter tuning and camera-motion constraints. This approach advances video editing by enabling interaction-rich, motion-consistent multi-object composition without model fine-tuning.

Abstract

Video composition is the core task of video editing. Although image composition based on diffusion models has been highly successful, it is not straightforward to extend the achievement to video object composition tasks, which not only exhibit corresponding interaction effects but also ensure that the objects in the composited video maintain motion and identity consistency, which is necessary to composite a physical harmony video. To address this challenge, we propose a Multiple Video Object Composition (MVOC) method based on diffusion models. Specifically, we first perform DDIM inversion on each video object to obtain the corresponding noise features. Secondly, we combine and edit each object by image editing methods to obtain the first frame of the composited video. Finally, we use the image-to-video generation model to composite the video with feature and attention injections in the Video Object Dependence Module, which is a training-free conditional guidance operation for video generation, and enables the coordination of features and attention maps between various objects that can be non-independent in the composited video. The final generative model not only constrains the objects in the generated video to be consistent with the original object motion and identity, but also introduces interaction effects between objects. Extensive experiments have demonstrated that the proposed method outperforms existing state-of-the-art approaches. Project page: https://sobeymil.github.io/mvoc.com.

Figures (14)

• Figure 1: Multiple Video Object Composition Results. Given multiple video objects (e.g. Background, Object1, Object2), our method enables presenting the interaction effects between multiple video objects and maintaining the motion and identity consistency of each object in the composited video.
• Figure 2: Multiple video object composition framework. Our method presents a two-stage approach: video object preprocessing and generative video editing. In the preprocessing stage, we perform DDIM inversion, object extraction and paste, as well as mask extraction. In the editing stage, we edit the first frame by an image editing model, then use video object dependence for conditional guidance video generation.
• Figure 3: Illustration of three video object composition.
• Figure 4: Qualitative comparisons. We utilize different methods to composite three video objects into one video. Our method is capable of not only maintaining the motion and identity consistency of each object but also presenting the interaction effects between multiple video objects in the composited video.
• Figure 5: Qualitative comparisons with SVC. Compared to SVC, our method performs better in generating interactive effects on objects, such as shadows.