Learning Stochastic Bridges for Video Object Removal via Video-to-Video Translation

TL;DR

The paper tackles the problem of removing objects from videos without sacrificing temporal coherence or background realism. It replaces diffusion-from-noise paradigms with a stochastic bridge that directly connects the source video latent $z_{\text{src}}$ to the target latent $z_{\text{tgt}}$ in a VP-SDE framework, using a frozen VAE for latent encoding and dual-role conditioning to preserve structure. A key contribution is the adaptive mask modulation (AMM), which dynamically relaxes the strong source prior in large-occlusion regions while maintaining fidelity elsewhere, enabling robust removal of objects of varying scales. Extensive experiments on synthetic (ROSE) and real-world composite datasets demonstrate improved visual quality and temporal stability over diffusion-based and other baseline methods, highlighting the practical potential for high-fidelity video editing with responsible deployment considerations.

Abstract

Existing video object removal methods predominantly rely on diffusion models following a noise-to-data paradigm, where generation starts from uninformative Gaussian noise. This approach discards the rich structural and contextual priors present in the original input video. Consequently, such methods often lack sufficient guidance, leading to incomplete object erasure or the synthesis of implausible content that conflicts with the scene's physical logic. In this paper, we reformulate video object removal as a video-to-video translation task via a stochastic bridge model. Unlike noise-initialized methods, our framework establishes a direct stochastic path from the source video (with objects) to the target video (objects removed). This bridge formulation effectively leverages the input video as a strong structural prior, guiding the model to perform precise removal while ensuring that the filled regions are logically consistent with the surrounding environment. To address the trade-off where strong bridge priors hinder the removal of large objects, we propose a novel adaptive mask modulation strategy. This mechanism dynamically modulates input embeddings based on mask characteristics, balancing background fidelity with generative flexibility. Extensive experiments demonstrate that our approach significantly outperforms existing methods in both visual quality and temporal consistency.

Figures (4)

• Figure 1: The framework of BridgeRemoval. The video inputs are projected into latent space using a frozen VAE. Unlike standard diffusion, we employ a VP-SDE Bridge formulation to interpolate a trajectory ($z_t$) directly from the source video prior ($z_{\text{src}}$) to the clean target ($z_{\text{tgt}}$). The DiT-based model is conditioned on text embeddings ($c_{\text{text}}$) and a spatial input formed by concatenating the mask ($z_{\text{M}}$) with the source latent ($z_{\text{src}}$), optimized via a velocity-matching objective ($\mathcal{L}_{\text{bridge}}$).
• Figure 2: Texture quality comparison with the state-of-the-art methods.
• Figure 3: Texture quality comparison with the state-of-the-art methods.
• Figure 4: Temporal consistency comparison with the state-of-the-art methods.