DreamInsert: Zero-Shot Image-to-Video Object Insertion from A Single Image

TL;DR

DreamInsert presents a training-free, two-stage framework for zero-shot image-to-video insertion of an object from a single image into a background video. By decomposing motion into coarse trajectory-driven motion creation and a subsequent spatiotemporal refinement via inversion-based editing, it achieves plausible unseen motion and coherent environmental interactions without end-to-end training. The approach leverages trajectory conditioning, segmentation-based object merging, Pixel and Latent Noise Injection, and Double Inversion to ensure fidelity and temporal consistency, demonstrated on the I2VIns dataset with strong quantitative and qualitative results. The work advances zero-shot video synthesis by combining diffusion-based generation with inversion-based refinement, enabling flexible object insertion in diverse scenes while highlighting remaining limitations and ethical considerations for realistic content creation.

Abstract

Recent developments in generative diffusion models have turned many dreams into realities. For video object insertion, existing methods typically require additional information, such as a reference video or a 3D asset of the object, to generate the synthetic motion. However, inserting an object from a single reference photo into a target background video remains an uncharted area due to the lack of unseen motion information. We propose DreamInsert, which achieves Image-to-Video Object Insertion in a training-free manner for the first time. By incorporating the trajectory of the object into consideration, DreamInsert can predict the unseen object movement, fuse it harmoniously with the background video, and generate the desired video seamlessly. More significantly, DreamInsert is both simple and effective, achieving zero-shot insertion without end-to-end training or additional fine-tuning on well-designed image-video data pairs. We demonstrated the effectiveness of DreamInsert through a variety of experiments. Leveraging this capability, we present the first results for Image-to-Video object insertion in a training-free manner, paving exciting new directions for future content creation and synthesis. The code will be released soon.

Figures (21)

• Figure 1: Visual examples on "Coffee-Bird", where DreamInsert realizes zero-shot insertion for static object into dynamic video.
• Figure 2: The overview of DreamInsert where $\mathtt{Merge}$ denotes the rescale + replace operation and $\mathbf{Z}^\text{merge} = \mathtt{Merge}(\mathbf{x}^{\text{obj}}, \mathbf{M}^{\text{merge}} )$, showcasing two stages: the blue part is the first stage of motion creation, while the purple part is the second stage of spatiotemporal alignment.
• Figure 3: Left: Pixel noise injection with the region illustration, where gray for the background, white for the object area and black for the interaction (IA) area. Right: The overview of Double Inversion pipeline with latent noise injection. We only add noise in the latent's IA area and obtain coarse frame after denoising.
• Figure 4: Existing training-based subject-driven method can hardly maintain consistency in scenarios of disordering semantic.
• Figure 5: Visual examples of outputs in two stages. Top: motion creation in the 1st stage; Bottom: alignment in the 2nd stage.