VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence

TL;DR

This work tackles video editing with shape changes by swapping the subject identity using semantic point correspondences, enabling target concepts to alter appearance while preserving background. VideoSwap integrates a motion layer and sparse semantic points, learning point embeddings and registrations to guide denoising and maintain temporal consistency, and supports both predefined and customized targets via ED-LoRA. It also offers interactive tools, including point removal and dragging with displacement propagation through Layered Neural Atlas, and demonstrates state-of-the-art results through comprehensive qualitative and quantitative evaluations. While effective, it acknowledges limitations in point tracking and canonical-space representation, and points toward faster, more interactive future editing workflows and safeguards against misuse.

Abstract

Current diffusion-based video editing primarily focuses on structure-preserved editing by utilizing various dense correspondences to ensure temporal consistency and motion alignment. However, these approaches are often ineffective when the target edit involves a shape change. To embark on video editing with shape change, we explore customized video subject swapping in this work, where we aim to replace the main subject in a source video with a target subject having a distinct identity and potentially different shape. In contrast to previous methods that rely on dense correspondences, we introduce the VideoSwap framework that exploits semantic point correspondences, inspired by our observation that only a small number of semantic points are necessary to align the subject's motion trajectory and modify its shape. We also introduce various user-point interactions (\eg, removing points and dragging points) to address various semantic point correspondence. Extensive experiments demonstrate state-of-the-art video subject swapping results across a variety of real-world videos.

Figures (10)

• Figure 1: Customized video subject swapping results with VideoSwap. VideoSwap supports shape change in the swapped results while aligning with the source motion trajectory. The swapped target can be either a predefined concept from a pretrained model (e.g., helicopter) or a customized concept (denoted by $V^*$). Previous methods based on implicit motion encoding and dense correspondence do not perform well in subject swapping with shape changes. We encourage readers to click and play the video clips in this figure using Adobe Acrobat.
• Figure 2: Customized video subject swapping results of VideoSwap on various concepts. The swapping target can either be a predefined concept in the pretrained model (e.g., helicopter) or a customized concept created by ED-LoRA gu2023mix (denoted as $V^*$). We encourage readers to click and play the video clips in this figure using Adobe Acrobat. For legal issues, we cannot display the human swap results.
• Figure 3: Toy experiment exploring semantic point correspondence. We encourage readers to click and play the video clips in this figure using Adobe Acrobat.
• Figure 4: Overview of the VideoSwap pipeline for customized video subject swapping.
• Figure 5: Pipelines for semantic point extraction (Sec. \ref{['sec:source_point_extract']}) and semantic point registration (Sec. \ref{['sec:source_point_register']}) in VideoSwap. In semantic point extraction, users define semantic points at a keyframe. We then extract the trajectory and embedding of those semantic points from the video. In semantic point registration, the semantic point embedding is projected by multiple 2-layer learnable MLPs, placed in empty features based on their coordinates, and then added element-wise to the diffusion model as motion guidance.