VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priors

TL;DR

VideoHandles tackles the challenge of editing 3D object compositions in videos by leveraging a trained flow-based video prior. It lifts intermediate features to a shared 3D reconstruction, applies a user-specified 3D transformation, and uses a warping-guided, energy-driven generative process to produce temporally coherent edits without training or fine-tuning. The method introduces 3D-aware warping, foreground/background energy terms, self-attention-based gradient weighting, and null-text prediction to preserve object identity while updating context like shadows and reflections. Comprehensive experiments on generated and real videos, plus a user study, demonstrate superior plausibility and temporal consistency relative to per-frame baselines and image-based editing approaches. This zero-shot framework offers practical, realistic video edits that align with user-specified 3D poses and can extend to various video generative models.

Abstract

Generative methods for image and video editing use generative models as priors to perform edits despite incomplete information, such as changing the composition of 3D objects shown in a single image. Recent methods have shown promising composition editing results in the image setting, but in the video setting, editing methods have focused on editing object's appearance and motion, or camera motion, and as a result, methods to edit object composition in videos are still missing. We propose \name as a method for editing 3D object compositions in videos of static scenes with camera motion. Our approach allows editing the 3D position of a 3D object across all frames of a video in a temporally consistent manner. This is achieved by lifting intermediate features of a generative model to a 3D reconstruction that is shared between all frames, editing the reconstruction, and projecting the features on the edited reconstruction back to each frame. To the best of our knowledge, this is the first generative approach to edit object compositions in videos. Our approach is simple and training-free, while outperforming state-of-the-art image editing baselines.

Figures (8)

• Figure 1: VideoHandles Architecture. We use the intermediate features $\Psi_{\mathrm{src}}$ of a video generative model to represent the identity of objects in a source video. Given a 3D transformation of an object, we can use a 3D reconstruction of the scene to warp the intermediate features consistently across frames. Guiding the video generator with these warped features $\Psi_{\mathrm{tgt}}$ gives us a an edited video where the object is transformed, while also maintaining the plausibility of effects like shadows and reflections.
• Figure 2: Visualization of our self-attention-based masks. The masks do not only include the the edited object, but also regions requiring semantic adjustments, such as a new reflection under the wine glass and newly disoccluded lamp.
• Figure 3: A qualitative comparison with other baselines. The examples show that ours best demonstrates plausibility by avoiding object duplication, adjusting shadows properly, and maintaining consistent outputs across frames, desipte warping errors, as illustrated in the direct frame warping outputs (column 2).
• Figure 4: User study results on the plausibility, identity preservation, and edit coherence of the edited videos. Each bar pair shows user preferences, with the green bar for our method and the other for the baseline, along with 95% confidence intervals. We also include a comparison with the input video to represent the upper bound of plausibility.
• Figure 5: A qualitative comparison of the ablation study. We show the effect of each component in our method. As demonstrated, our full method avoids object duplication and unnecessary drastic changes in the background, while effectively preserving the identity of the selected object.