MovieCharacter: A Tuning-Free Framework for Controllable Character Video Synthesis

TL;DR

MovieCharacter presents a tuning-free, modular framework for controllable character video synthesis that bypasses heavy fine-tuning or proprietary datasets. It decomposes the task into segmentation/tracking, object removal, motion imitation, and composition, integrating lighting-aware harmonization and edge-aware refinement to achieve coherent, realistic results. By leveraging open-source models (e.g., SAM2, ProPainter, PCT-Net) and pose-guided diffusion techniques, the approach emphasizes accessibility, efficiency, and broad applicability in media production and interactive contexts. The work demonstrates competitive quality across cinematic scenes with ablations confirming the value of the composition components, while acknowledging limitations in handling complex interactions and occlusions.

Abstract

Recent advancements in character video synthesis still depend on extensive fine-tuning or complex 3D modeling processes, which can restrict accessibility and hinder real-time applicability. To address these challenges, we propose a simple yet effective tuning-free framework for character video synthesis, named MovieCharacter, designed to streamline the synthesis process while ensuring high-quality outcomes. Our framework decomposes the synthesis task into distinct, manageable modules: character segmentation and tracking, video object removal, character motion imitation, and video composition. This modular design not only facilitates flexible customization but also ensures that each component operates collaboratively to effectively meet user needs. By leveraging existing open-source models and integrating well-established techniques, MovieCharacter achieves impressive synthesis results without necessitating substantial resources or proprietary datasets. Experimental results demonstrate that our framework enhances the efficiency, accessibility, and adaptability of character video synthesis, paving the way for broader creative and interactive applications.

Figures (5)

• Figure 1: MovieCharacter enables the replacement of any character in a movie with a 2D reference character, facilitating the synthesis of customized animated avatars. By utilizing driving motions sourced from movies, MovieCharacter can generate movements that closely mimic the original character's actions. Additionally, our framework allows for the seamless integration of cinematic scenes, ensuring natural interactions between the synthesized characters and their environments.
• Figure 2: The overall architecture of MovieCharacter, where the top part is the pipeline for detecting and removing the targeted character from the video, and the bottom part is character motion imitation branch. The composition module serves to integrate the outputs from the upper and lower branches of the framework.
• Figure 3: Video object removal is necessary for our proposed MovieCharacter. The resultant frames at different time steps (column-wise) are presented with different operations (row-wise). (a) The original video clip. (b) The tracked segmentation (green color) of the object that is expected to be changed. (c) Video clip after applying video object removal. (d) Composition of the new character and the video clip without object removal. (e) Composition with object removal.
• Figure 4: Compositing a character with a video clip requires further refinements. Frames at different time steps (column-wise) are displayed with different refinements (row-wise). (a) The original video clip. (b) Direct composition without refinements. A green dashed-line bounding box marks the new character. (c) Composition after video harmonization. (d) Composition overlapped with the segmentation (green color) of the edge area. (e) Composition with refined edges.
• Figure 5: Examples of synthesizing avatar animations across multiple cinematic contexts, utilizing various reference characters.