MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models

TL;DR

MoTrans tackles the challenge of customizing motion transfer from reference videos to new subjects in varied scenes by decoupling appearance from motion through a two-stage diffusion-based framework. It leverages an MLLM-based recaptioner to expand reference prompts for appearance and introduces an appearance injector along with a motion-specific embedding to emphasize motion in temporal modeling, using calibrated LoRA adapters. The approach achieves strong motion fidelity, text/appearance alignment, and temporal consistency, outperforming state-of-the-art one-shot and few-shot customization methods and enabling simultaneous subject and motion customization. While effective, it is currently tailored to short clips and limb-based human motions, with future work targeting longer sequences and broader action types.

Abstract

Existing pretrained text-to-video (T2V) models have demonstrated impressive abilities in generating realistic videos with basic motion or camera movement. However, these models exhibit significant limitations when generating intricate, human-centric motions. Current efforts primarily focus on fine-tuning models on a small set of videos containing a specific motion. They often fail to effectively decouple motion and the appearance in the limited reference videos, thereby weakening the modeling capability of motion patterns. To this end, we propose MoTrans, a customized motion transfer method enabling video generation of similar motion in new context. Specifically, we introduce a multimodal large language model (MLLM)-based recaptioner to expand the initial prompt to focus more on appearance and an appearance injection module to adapt appearance prior from video frames to the motion modeling process. These complementary multimodal representations from recaptioned prompt and video frames promote the modeling of appearance and facilitate the decoupling of appearance and motion. In addition, we devise a motion-specific embedding for further enhancing the modeling of the specific motion. Experimental results demonstrate that our method effectively learns specific motion pattern from singular or multiple reference videos, performing favorably against existing methods in customized video generation.

Figures (13)

• Figure 1: Overview of the proposed MoTrans. In the appearance learning stage, an MLLM-based recaptioner is employed to extend the base prompt, encouraging the spatial LoRAs to sufficiently learn appearance information. The weights of spatial LoRAs are shared in the second stage. In the motion learning stage, video frame embeddings are injected as appearance priors, compelling the temporal LoRAs to concentrate on motion learning. Furthermore, we adopt MLP to learn a motion-specific embedding, which is jointly trained with the temporal LoRAs to fit specific motion patterns in the reference video.
• Figure 2: Illustration of multimodal recaptioning. Given an image, an MLLM-based recaptioner is employed to expand the base prompt according to the task instruction, enabling the extended prompt to fully describe its appearance.
• Figure 3: Details of trainable LoRAs and appearance injector. (a) Parameters of the base model are frozen and only parameters of LoRAs are updated. (b) The image embedding is processed through a Linear layer before being fused with the hidden states from the spatial transformers. This pre-injected appearance prior encourages the temporal LoRAs to capture motion patterns effectively.
• Figure 4: Qualitative comparison of customized motion transfer. The reference videos on the left demonstrate the motion of a person slowly lifting their hand to drink water. On the right, the videos show a skateboarding pushing action, where the person pushes off the ground with their foot and then slides forward. For one-shot motion customization, the learned motion refers to the second example from the reference videos. Best viewed zoomed-in.
• Figure 5: User study. For each metric, the percentages attributed to all methods sum to 1. MoTrans accounts for the largest proportion, indicating that the videos generated by our method exhibit superior text alignment, temporal consistency, and the closest resemblance to the reference video.