DirectSwap: Mask-Free Cross-Identity Training and Benchmarking for Expression-Consistent Video Head Swapping

TL;DR

The paper tackles mask-free, cross-identity video head swapping by introducing HeadSwapBench, a paired dataset generated via a video-editing pipeline to align motion with target identities. It then proposes DirectSwap, a diffusion-based framework with a dual-canvas conditioning scheme and the MEAR loss to jointly optimize identity transfer and temporal coherence without masking. Empirical results show state-of-the-art performance in identity fidelity, pose/expression consistency, and temporal stability on HeadSwapBench and in-the-wild videos, with ablations confirming the benefits of unmasked learning and MEAR. The work paves a path toward more realistic, geometry-aware digital humans and provides publicly releasable data and code to spur further research.

Abstract

Video head swapping aims to replace the entire head of a video subject, including facial identity, head shape, and hairstyle, with that of a reference image, while preserving the target body, background, and motion dynamics. Due to the lack of ground-truth paired swapping data, prior methods typically train on cross-frame pairs of the same person within a video and rely on mask-based inpainting to mitigate identity leakage. Beyond potential boundary artifacts, this paradigm struggles to recover essential cues occluded by the mask, such as facial pose, expressions, and motion dynamics. To address these issues, we prompt a video editing model to synthesize new heads for existing videos as fake swapping inputs, while maintaining frame-synchronized facial poses and expressions. This yields HeadSwapBench, the first cross-identity paired dataset for video head swapping, which supports both training (\TrainNum{} videos) and benchmarking (\TestNum{} videos) with genuine outputs. Leveraging this paired supervision, we propose DirectSwap, a mask-free, direct video head-swapping framework that extends an image U-Net into a video diffusion model with a motion module and conditioning inputs. Furthermore, we introduce the Motion- and Expression-Aware Reconstruction (MEAR) loss, which reweights the diffusion loss per pixel using frame-difference magnitudes and facial-landmark proximity, thereby enhancing cross-frame coherence in motion and expressions. Extensive experiments demonstrate that DirectSwap achieves state-of-the-art visual quality, identity fidelity, and motion and expression consistency across diverse in-the-wild video scenes. We will release the source code and the HeadSwapBench dataset to facilitate future research.

Figures (6)

• Figure 1: Qualitative results on zero-shot video head swapping. Given a reference head and a driving video (Vd), our method synthesizes the entire head while preserving the background, body, and expression dynamics. Top: comparison with existing methods on HeadSwapBench with ground-truth videos ($V_a$). Bottom: our temporally consistent results across video frames.
• Figure 2: HeadSwapBench dataset generation and training pairs. Left: pipeline of prompting VACE vace to synthesize new heads for existing videos ($V_a$), resulting in fake head-swapped driven video ($V_d$), while maintaining frame-synchronized facial poses and expressions. Right: example training pairs where input $V_d$ and ground truth $V_a$ share background and motion but differ in identity. See details in Sec. \ref{['sec:dataset1']}.
• Figure 3: Overview of the DirectSwap framework for mask-free video head swapping. Left: Dual-canvas module arranges motion and identity in spatially aligned latent canvases, enabling geometry-consistent identity transport in motion enhanced U-Net model. Right: Motion- and Expression-Aware Reconstruction (MEAR) adaptively reweights supervision using motion and expression cues to improve perceptual fidelity and temporal coherence.
• Figure 4: Comparison with prior methods. DiffSwap zhao2023diffswap and InSwapper haofanwang are face-swapping methods limited to the facial region, while IP-Adapter ye2023ip-adapter, HeadSwap lesliezhoa, and VACE vacewan2025 perform full head swapping. The first two rows show results on our test set, and the remaining rows present in-the-wild examples (where ground truth is unavailable). Our method delivers sharper geometry, stronger identity preservation, and more faithful expression transfer with superior temporal coherence.
• Figure 5: Impact of training strategies. Head masking distorts geometry, while rectangular masking weakens pose, expression, and background fidelity. Our unmasked training preserves both.