Anything in Any Scene: Photorealistic Video Object Insertion

TL;DR

The paper tackles the challenge of photorealistic video content creation by enabling insertion of arbitrary objects into real dynamic scenes while preserving geometric realism, lighting fidelity, and photorealism. It introduces Anything in Any Scene, a modular pipeline with (i) object placement and stabilization, (ii) HDR-based lighting and shadow generation, and (iii) a coarse-to-fine style transfer module, supported by a scalable assets bank and retrieval system. Extensive experiments on PandaSet and ScanNet++ show improvements in FID and human realism scores over baselines, and downstream validation on CODA demonstrates augmented data benefits for rare-object detection. The framework offers a cost-effective, flexible platform for data augmentation, virtual reality, and video editing, with clear avenues for integrating improved submodules and assets.

Abstract

Realistic video simulation has shown significant potential across diverse applications, from virtual reality to film production. This is particularly true for scenarios where capturing videos in real-world settings is either impractical or expensive. Existing approaches in video simulation often fail to accurately model the lighting environment, represent the object geometry, or achieve high levels of photorealism. In this paper, we propose Anything in Any Scene, a novel and generic framework for realistic video simulation that seamlessly inserts any object into an existing dynamic video with a strong emphasis on physical realism. Our proposed general framework encompasses three key processes: 1) integrating a realistic object into a given scene video with proper placement to ensure geometric realism; 2) estimating the sky and environmental lighting distribution and simulating realistic shadows to enhance the light realism; 3) employing a style transfer network that refines the final video output to maximize photorealism. We experimentally demonstrate that Anything in Any Scene framework produces simulated videos of great geometric realism, lighting realism, and photorealism. By significantly mitigating the challenges associated with video data generation, our framework offers an efficient and cost-effective solution for acquiring high-quality videos. Furthermore, its applications extend well beyond video data augmentation, showing promising potential in virtual reality, video editing, and various other video-centric applications. Please check our project website https://anythinginanyscene.github.io for access to our project code and more high-resolution video results.

Figures (18)

• Figure 1: Examples of simulated video frame with wrong lighting environment estimation, false object placement position, and unrealistic texture style, which make the image lack physical realism
• Figure 2: Overview of proposed Anything in Any Scene framework for photorealistic video object insertion
• Figure 3: Example of driving scene video for object placement. The red point in each image is the location for object insertion.
• Figure 4: Examples of original sky image, reconstructed HDR image, and its associated sun lighting distribution map
• Figure 5: Examples of Original and Reconstructed HDR Environmental Panoramic Image