MotionAdapter: Video Motion Transfer via Content-Aware Attention Customization

TL;DR

MotionAdapter tackles the challenge of transferring complex motions between videos by disentangling motion from appearance through analysis of 3D full-attention in DiT-based T2V models and then customizing the motion with semantic correspondences derived from DINO. The framework separates foreground and background motion, warps motion fields via content-aware correspondences, and refines them before guiding the DiT denoising process, enabling accurate and semantically aligned motion transfer even across large content gaps. Extensive experiments on a DAVIS-based subset show that MotionAdapter outperforms state-of-the-art methods in both video quality and motion fidelity, and it supports flexible motion editing such as zooming. The approach advances robust motion transfer for diffusion-based video generation and offers practical benefits for editing and content creation in complex scenes.

Abstract

Recent advances in diffusion-based text-to-video models, particularly those built on the diffusion transformer architecture, have achieved remarkable progress in generating high-quality and temporally coherent videos. However, transferring complex motions between videos remains challenging. In this work, we present MotionAdapter, a content-aware motion transfer framework that enables robust and semantically aligned motion transfer within DiT-based T2V models. Our key insight is that effective motion transfer requires \romannumeral1) explicit disentanglement of motion from appearance and \romannumeral 2) adaptive customization of motion to target content. MotionAdapter first isolates motion by analyzing cross-frame attention within 3D full-attention modules to extract attention-derived motion fields. To bridge the semantic gap between reference and target videos, we further introduce a DINO-guided motion customization module that rearranges and refines motion fields based on content correspondences. The customized motion field is then used to guide the DiT denoising process, ensuring that the synthesized video inherits the reference motion while preserving target appearance and semantics. Extensive experiments demonstrate that MotionAdapter outperforms state-of-the-art methods in both qualitative and quantitative evaluations. Moreover, MotionAdapter naturally supports complex motion transfer and motion editing tasks such as zooming.

Figures (9)

• Figure 1: Overview of our MotionAdapter. Given a reference video $\mathcal{V}_{ref}$, we first encode it to latent representations $z_0$, and pass to the T2V to get the full attention map $\mathcal{A}$. We then extract the motion in the cross-frame attention by seeking nearest Top-$K$ pixels. By analyzing temporal correspondences in cross-frame attention maps, we derive the attention motion$\mathcal{M}_{ref}$ that disentangles motion from appearance. For custom the motion that is compatible with the target context, we introduce the content-aware motion customization. We compute a semantic correspondence between the reference and target content using DINO features, and customize the motion field accordingly to obtain $\mathcal{M}_{cust}$, composed of foreground and background motion. Finally, Gaussian smoothing refines the customized motion into $\mathcal{M}_{final}$, which is used to guide motion transfer.
• Figure 2: We plot the MSE distance between GT flow and cross-frame attention motions extracted from various DiT blocks and timesteps, we can see that attention motions extracted from mid-level DiT blocks and lower noise timesteps gain the lower distance.
• Figure 3: We visualize the attention motion field obtained from various noise steps and attention blocks, and the motion derived from $t=5,b=18$ is more simarity with the GT optical flow.
• Figure 4: The attention extracted from reference video contains the reference-specific shape information, resulting in the geometric distortions in the target videos (see in red box).
• Figure 5: Qualitative comparison of motion transfer methods. Our MotionAdapter enables robust, content-aware motion transfer, producing temporally coherent and semantically aligned videos that preserve both reference motion and target appearance, even under large semantic gaps and complex scenarios.