Object-WIPER : Training-Free Object and Associated Effect Removal in Videos

TL;DR

Object-WIPER introduces a training-free framework for removing both a target object and its associated effects from videos by leveraging cross-attention and self-attention within a pre-trained text-to-video diffusion model. The method locates associated effects via a two-stage localization (text-guided proposal plus self-attention refinement), then performs latent inversion and denoising with timestep-adaptive masking and attention scaling, including Gaussian reinitialization of the removed region. A new evaluation metric, TokSim, assesses temporal consistency, background coherence, and complete removal, and is complemented by a real-world benchmark, WIPER-Bench, plus DAVIS experiments. Results show Object-WIPER surpasses both training-based and training-free baselines on TokSim while maintaining competitive background fidelity, enabling high-quality, training-free object removal in real-world videos. The work also demonstrates strong human agreement with TokSim and provides a detailed ablation study to justify each component of the pipeline.

Abstract

In this paper, we introduce Object-WIPER, a training-free framework for removing dynamic objects and their associated visual effects from videos, and inpainting them with semantically consistent and temporally coherent content. Our approach leverages a pre-trained text-to-video diffusion transformer (DiT). Given an input video, a user-provided object mask, and query tokens describing the target object and its effects, we localize relevant visual tokens via visual-text cross-attention and visual self-attention. This produces an intermediate effect mask that we fuse with the user mask to obtain a final foreground token mask to replace. We first invert the video through the DiT to obtain structured noise, then reinitialize the masked tokens with Gaussian noise while preserving background tokens. During denoising, we copy values for the background tokens saved during inversion to maintain scene fidelity. To address the lack of suitable evaluation, we introduce a new object removal metric that rewards temporal consistency among foreground tokens across consecutive frames, coherence between foreground and background tokens within each frame, and dissimilarity between the input and output foreground tokens. Experiments on DAVIS and a newly curated real-world associated effect benchmark (WIPER-Bench) show that Object-WIPER surpasses both training-based and training-free baselines in terms of the metric, achieving clean removal and temporally stable reconstruction without any retraining. Our new benchmark, source code, and pre-trained models will be publicly available.

Figures (14)

• Figure 2: Associated effects localization. The figure shows the processing of the video latents to obtain the mask for the object and the associated objects using the cross attention maps and the self attention maps. First, through the cross-attention scores, we obtain the patches of interest that are highly correlated with the query text tokens. Further, through self-attention scores, we identify the tokens that have the highest response to these patches of interest to obtain the final mask.
• Figure 3: The figure shows all parts of our object removal algorithm once the mask for the associated effects algorithm is obtained. We perform inversion of video latent using RF Solver Edit while saving the background values for several iterations. The inverted noise is reinitialised in the mask region and is denoised with copying back the background values to obtain the output video.
• Figure 4: Timestep adaptive masking. During inversion, the object’s footprint expands as noise increases, causing fixed masks to leak object tokens while denoising copying. In contrast, adaptive masks augmented with associated-effect regions prevent such leakage and enable complete removal of the object and its effects.
• Figure 5: The proposed metric, TokSim scores very high only when the object is fully removed and progressively becomes lower as the object removal reduces. For VAE-reconstruction where the object is not removed at all, the TokSim is nearly zero. However, the ranges of the values for BG-PSNR and video quality across the vastly different outputs are extremely compressed and do not serve the purpose of unambiguously distinguishing between the object removal approaches of varied capabilities.
• Figure 6: Qualitative comparison between our method and existing approaches on (left) WIPER-Bench and (right) DAVIS. On WIPER-Bench, our method removes both the object and its associated effects across diverse scenarios, whereas both training-free and training-based baselines fail to remove the object completely. On DAVIS, our method achieves full object removal; notably, in the car example (third column), even training-based methods such as Gen-Prop and ROSE are unable to do so.