End-to-End Video Character Replacement without Structural Guidance

TL;DR

MoCha tackles end-to-end video character replacement with only a single frame mask, avoiding per-frame segmentation and explicit structural priors. It combines in-context learning with a condition-aware RoPE and an RL-based post-training stage to transfer motion and facial expressions from a source to a reference identity while preserving background and lighting. A comprehensive data pipeline—comprising Unreal Engine rendered data, expression-driven animation data, and augmented video-mask data—enables robust training without paired, framewise supervision. Empirical results show MoCha outperforms state-of-the-art baselines in identity preservation, temporal coherence, and facial fidelity, demonstrating strong generalization to complex scenes and multi-character interactions. The work offers practical impact for film/post-production, personalized avatars, and broader subject replacement tasks, with potential extensions to face swapping and virtual try-on.

Abstract

Controllable video character replacement with a user-provided identity remains a challenging problem due to the lack of paired video data. Prior works have predominantly relied on a reconstruction-based paradigm that requires per-frame segmentation masks and explicit structural guidance (e.g., skeleton, depth). This reliance, however, severely limits their generalizability in complex scenarios involving occlusions, character-object interactions, unusual poses, or challenging illumination, often leading to visual artifacts and temporal inconsistencies. In this paper, we propose MoCha, a pioneering framework that bypasses these limitations by requiring only a single arbitrary frame mask. To effectively adapt the multi-modal input condition and enhance facial identity, we introduce a condition-aware RoPE and employ an RL-based post-training stage. Furthermore, to overcome the scarcity of qualified paired-training data, we propose a comprehensive data construction pipeline. Specifically, we design three specialized datasets: a high-fidelity rendered dataset built with Unreal Engine 5 (UE5), an expression-driven dataset synthesized by current portrait animation techniques, and an augmented dataset derived from existing video-mask pairs. Extensive experiments demonstrate that our method substantially outperforms existing state-of-the-art approaches. We will release the code to facilitate further research. Please refer to our project page for more details: orange-3dv-team.github.io/MoCha

Figures (9)

• Figure 1: Examples generated by MoCha. MoCha enables high-fidelity character replacement in source videos based on provided reference images for diverse subjects, including virtual (first row) and real-human (second row) characters. Furthermore, our approach robustly preserves original lighting conditions (third row) and effectively handles multi-character occlusion and interaction (fourth row).
• Figure 2: (a) Reconstruction-based paradigm used in baseline methods. (b) Non-reconstruction-based paradigm used in MoCha.
• Figure 3: Overview of MoCha. Training MoCha consists of two stages: (a) In-Context Learning. We apply the condition-aware RoPE to the concatenated tokens and train the DiT backbone. (b) Post-Training. We employ an RL-based strategy to further enhance the facial consistency.
• Figure 4: Overview of the data construction pipeline. We propose three methods to construct the training dataset: (I) Rendered data built with Unreal Engine 5. (II) Expression-driven face animation data generated through portrait animation methods. (III) Augmented data synthesized from traditional video-mask pairs.
• Figure 5: Comparison with state-of-the-art methods. The results show that MoCha can replace the character with more consistent animation, higher facial expressiveness, and more natural lighting effects.