InsertAnywhere: Bridging 4D Scene Geometry and Diffusion Models for Realistic Video Object Insertion

TL;DR

This work tackles realistic video object insertion by jointly modeling 4D scene geometry and appearance via a two-stage framework. A 4D-aware mask generation module reconstructs scene structure, propagates user-specified object placement, and yields temporally coherent masks, while a diffusion-based video generator renders the inserted object with accurate local lighting. ROSE++ provides illumination-aware supervision by transforming object removal data into insertion triplets using VLM-derived references, enabling photometric consistency. The method, trained with LoRA on a large video diffusion model and evaluated on a dedicated VOIBench, achieves superior geometric alignment, occlusion handling, and lighting realism compared to commercial baselines. This approach has strong implications for commercial VOI, virtual product placement, and realistic video editing.

Abstract

Recent advances in diffusion-based video generation have opened new possibilities for controllable video editing, yet realistic video object insertion (VOI) remains challenging due to limited 4D scene understanding and inadequate handling of occlusion and lighting effects. We present InsertAnywhere, a new VOI framework that achieves geometrically consistent object placement and appearance-faithful video synthesis. Our method begins with a 4D aware mask generation module that reconstructs the scene geometry and propagates user specified object placement across frames while maintaining temporal coherence and occlusion consistency. Building upon this spatial foundation, we extend a diffusion based video generation model to jointly synthesize the inserted object and its surrounding local variations such as illumination and shading. To enable supervised training, we introduce ROSE++, an illumination aware synthetic dataset constructed by transforming the ROSE object removal dataset into triplets of object removed video, object present video, and a VLM generated reference image. Through extensive experiments, we demonstrate that our framework produces geometrically plausible and visually coherent object insertions across diverse real world scenarios, significantly outperforming existing research and commercial models.

Figures (13)

• Figure 1: InsertAnywhere enables realistic video object insertion by combining 4D scene understanding with diffusion-based video generation. Given user-specified control and reference objects, our method produces geometrically aligned and temporally consistent insertions across complex motions and viewpoints.
• Figure 2: InsertAnywhere is a two-stage VOI framework that uses 4D scene reconstruction to generate a user controllable and geometrically consistent mask, which then conditions a diffusion model fine-tuned on ROSE++ to synthesize illumination-aware object insertions.
• Figure 3: ROSE++ dataset generation pipeline: reference images are produced by the VLM-based object retrieval process. achiam2023gpt
• Figure 4: Qualitative comparison between closed-source commercial video generative tools and ours. Best viewed when zoomed in.
• Figure 5: Ablation results. We performed an ablation study by sequentially adding each of the proposed components. Best viewed when zoomed in.