MagicFight: Personalized Martial Arts Combat Video Generation

TL;DR

MagicFight tackles the gap in personalized two-person combat video generation by extending diffusion-based methods with a two-ID conditioning and pose-guided control. It introduces KungFu-Fiesta, a Unity-derived dataset, and a Multi-Modal Personalized Network featuring ID-attn, ReferenceNet, and a Background Crafter to maintain identity and background consistency across long combat sequences. The approach uses a Mixture Data Finetuning strategy and a two-stage finetuning regime to adapt to two-person interactions, along with clip fusion for arbitrary-length videos and a body-shape adaptive strategy to align generated bodies with reference IDs. Experimental results on reconstruction and open-set benchmarks, plus ablations, show improved identity preservation, pose accuracy, and visual quality, highlighting the system’s potential for realistic, controllable interactive martial arts video content generation.

Abstract

Amid the surge in generic text-to-video generation, the field of personalized human video generation has witnessed notable advancements, primarily concentrated on single-person scenarios. However, to our knowledge, the domain of two-person interactions, particularly in the context of martial arts combat, remains uncharted. We identify a significant gap: existing models for single-person dancing generation prove insufficient for capturing the subtleties and complexities of two engaged fighters, resulting in challenges such as identity confusion, anomalous limbs, and action mismatches. To address this, we introduce a pioneering new task, Personalized Martial Arts Combat Video Generation. Our approach, MagicFight, is specifically crafted to overcome these hurdles. Given this pioneering task, we face a lack of appropriate datasets. Thus, we generate a bespoke dataset using the game physics engine Unity, meticulously crafting a multitude of 3D characters, martial arts moves, and scenes designed to represent the diversity of combat. MagicFight refines and adapts existing models and strategies to generate high-fidelity two-person combat videos that maintain individual identities and ensure seamless, coherent action sequences, thereby laying the groundwork for future innovations in the realm of interactive video content creation. Website: https://MingfuYAN.github.io/MagicFight/ Dataset: https://huggingface.co/datasets/MingfuYAN/KungFu-Fiesta

Figures (6)

• Figure 1: Our MagicFight has 4 conditions for combat video generation, two reference IDs, a text prompt, a background image, and pose maps. The action of each frame is controlled by Pose Guider. The two IDs are personalized by our Personalized Attention (ID-attn) layer which can generate the respective appearance to the desired place. The user can provide a simple background image or use a pure white background and then generate a complex and reasonable background by our Background Crafter. With the long video generation technique, we can make arbitrary long videos (typically 10 seconds in our test).
• Figure 2: The body size adaptive strategy during inference.
• Figure 3: The results on two benchmarks. These solo dance models exhibit missing body parts and wrong actions, and they cannot be conditioned on background or generate background by prompt. Our MagicFight significantly mitigates these issues.
• Figure 4: The MagicFight results in open-set combat generation with smooth movements and consistent IDs. Because of page limits, we give more results in our appendix.
• Figure 5: Ablation Study. 1) ID appearance is ensured with sufficient IDs in the training data. 2) Adequate action in the training set is helpful. 3) Currently, end-to-end way struggles to handle complex backgrounds.