CoMo: Compositional Motion Customization for Text-to-Video Generation

TL;DR

CoMo tackles compositional motion customization in text-to-video generation by first decoupling appearance and motion via per-motion static and dynamic LoRAs learned in a phasewise manner, then composing multiple motions with a training-free divide-and-merge strategy that isolates regional motion influence during denoising. A MuDI-inspired Crop-and-Compare metric captures both per-motion fidelity and blending between motions, and a new benchmark supports systematic evaluation of multi-motion scenarios. Empirical results show state-of-the-art performance in both single-motion customization and multi-motion composition on a DiT-based diffusion backbone, with strong qualitative and quantitative gains and favorable human judgments. This work enables richer, more controllable video content without retraining for each new combination of motions, with practical implications for flexible video generation and editing.

Abstract

While recent text-to-video models excel at generating diverse scenes, they struggle with precise motion control, particularly for complex, multi-subject motions. Although methods for single-motion customization have been developed to address this gap, they fail in compositional scenarios due to two primary challenges: motion-appearance entanglement and ineffective multi-motion blending. This paper introduces CoMo, a novel framework for $\textbf{compositional motion customization}$ in text-to-video generation, enabling the synthesis of multiple, distinct motions within a single video. CoMo addresses these issues through a two-phase approach. First, in the single-motion learning phase, a static-dynamic decoupled tuning paradigm disentangles motion from appearance to learn a motion-specific module. Second, in the multi-motion composition phase, a plug-and-play divide-and-merge strategy composes these learned motions without additional training by spatially isolating their influence during the denoising process. To facilitate research in this new domain, we also introduce a new benchmark and a novel evaluation metric designed to assess multi-motion fidelity and blending. Extensive experiments demonstrate that CoMo achieves state-of-the-art performance, significantly advancing the capabilities of controllable video generation. Our project page is at https://como6.github.io/.

Figures (21)

• Figure 1: Our method CoMo enables learning and composing motions for text-to-video generation. The results demonstrate CoMo's effectiveness in: (a) single-motion customization, where a learned motion is transferred to a new subject and scene (e.g., a lion performing a burpee in an office); and (b) multi-motion composition, where two distinct, learned motions are performed by different subjects simultaneously within the same scene.
• Figure 2: Comparison of motion blending capabilities. We evaluate CoMo's ability to compose two distinct motions from different reference videos into a single scene with new subjects. Compared to baseline methods (e.g., linear merging and joint training) have incomplete and distorted movements, CoMO successfully generates a coherent video where both actions are performed naturally and simultaneously, preserving the integrity of each motion.
• Figure 2: Quantitative ablation.
• Figure 3: An overview of our proposed CoMo framework. CoMo consists of two phases: (1) decoupled single-motion learning and (2) plug-and-play multi-motion composition. In Phase 1, we first train a static LoRA module on random frames to learn the appearance of the reference video. Then, we freeze the static LoRA and train a dynamic LoRA module on the complete video to exclusively capture its motion patterns. In Phase 2, we introduce a divide-and-merge strategy for compositional motion generation, while all the weights are frozen.
• Figure 3: Inference Time with more motions.