Place Anything into Any Video
Ziling Liu, Jinyu Yang, Mingqi Gao, Feng Zheng
TL;DR
This work addresses the challenge of placing arbitrary objects into arbitrary videos with unknown camera parameters by introducing Place-Anything, a 3D-first editing pipeline comprising 3D model generation, video reconstruction, and 3D target insertion. The 3D model is generated from a photo or text via a diffusion-supervised 3D Gaussian mesh approach with mesh extraction, while video reconstruction estimates intrinsics, poses, and dense depth using a self-calibrating weighted bundle adjustment aided by optical flow. The 3D target insertion back-projects 2D user input into 3D coordinates, fits a placement plane with RANSAC, and renders multi-view insertions with PyTorch3D before compositing into the video; a key embedding step is the back-projection $[x_i,y_i,z_i] = \mathbf{G}_i \otimes \boldsymbol{\pi}^{-1}(\mathbf{u}_i, z_i, \boldsymbol{\theta})$. The system demonstrates high-fidelity, texture-consistent insertions across varied footage, enabling accessible applications in advertising, content creation, VR/AR, and post-production without requiring extensive 3D or rendering expertise.
Abstract
Controllable video editing has demonstrated remarkable potential across diverse applications, particularly in scenarios where capturing or re-capturing real-world videos is either impractical or costly. This paper introduces a novel and efficient system named Place-Anything, which facilitates the insertion of any object into any video solely based on a picture or text description of the target object or element. The system comprises three modules: 3D generation, video reconstruction, and 3D target insertion. This integrated approach offers an efficient and effective solution for producing and editing high-quality videos by seamlessly inserting realistic objects. Through a user study, we demonstrate that our system can effortlessly place any object into any video using just a photograph of the object. Our demo video can be found at https://youtu.be/afXqgLLRnTE. Please also visit our project page https://place-anything.github.io to get access.