ObjectMover: Generative Object Movement with Video Prior

TL;DR

ObjectMover reframes object movement within a single image as a sequence-to-sequence problem and leverages a pre-trained video diffusion prior to enforce consistent object identity and scene lighting across frames. To overcome data scarcity, it introduces a game-engine–based synthetic data pipeline and a multi-task learning strategy that also utilizes real video data via an auxiliary mask-based insertion task. The approach achieves state-of-the-art results for movement, removal, and insertion, demonstrated on curated ObjMove-A/B datasets with strong quantitative metrics and favorable human judgments. This work enables realistic, illumination-consistent object relocation in complex scenes and offers a practical framework for related image-editing tasks, with potential applications in photography, content creation, and visual effects.

Abstract

Simple as it seems, moving an object to another location within an image is, in fact, a challenging image-editing task that requires re-harmonizing the lighting, adjusting the pose based on perspective, accurately filling occluded regions, and ensuring coherent synchronization of shadows and reflections while maintaining the object identity. In this paper, we present ObjectMover, a generative model that can perform object movement in highly challenging scenes. Our key insight is that we model this task as a sequence-to-sequence problem and fine-tune a video generation model to leverage its knowledge of consistent object generation across video frames. We show that with this approach, our model is able to adjust to complex real-world scenarios, handling extreme lighting harmonization and object effect movement. As large-scale data for object movement are unavailable, we construct a data generation pipeline using a modern game engine to synthesize high-quality data pairs. We further propose a multi-task learning strategy that enables training on real-world video data to improve the model generalization. Through extensive experiments, we demonstrate that ObjectMover achieves outstanding results and adapts well to real-world scenarios.

Figures (25)

• Figure 1: Results of object movement. We demonstrate the object movement capability of ObjectMover in a variety of complex and challenging scenarios. ObjectMover can well keep object identity, synchronously edit lighting and shadow effects, complete occluded parts, understand materials, adjust object perspective, and comprehend occlusion relationships to generate realistic images where the object has been moved, maintaining all other elements of the scene unchanged.
• Figure 2: Limitations of existing methods for object movement. (Top): The approach of removing an object powerpaint and then re-inserting anydoor it for repositioning leads to issues with identity preservation and ineffective synchronization of lighting effects. (Bottom): The copy-paste-based method magic_fixup is unable to modify the perspective of the object.
• Figure 3: Results trained with different priors. The model fine-tuned from a video prior outperforms the one from an image prior.
• Figure 4: Model architecture. (Left): Our overall sequence-to-sequence framework by leveraging an image-to-video prior for training our single-frame image editing task (\ref{['sec:3.1']}). (Right): Frame formulation on different tasks for multi-task learning (\ref{['sec:3.3']}).
• Figure 5: Dataset comparison. Row (1-2) shows our synthetic data using a game engine; Row (3) shows existing synthetic data 3dit. Our data is more realistic and has complex lighting effects.