FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarios

TL;DR

FlexiAct tackles the problem of transferring actions across heterogeneous subjects by relaxing rigid spatial constraints. It introduces RefAdapter, an image-conditioned adapter with LoRA-based injection to enable spatial-structure adaptation and appearance preservation, and Frequency-aware Action Extraction (FAE), which extracts and encodes action signals during diffusion denoising by modulating attention across timesteps. The approach extends CogVideoX-I2V with a two-stage training pipeline and demonstrates strong performance on diverse subjects and domains, outperforming global-motion baselines and pose-based methods in both motion fidelity and appearance consistency. The results indicate FlexiAct enables flexible, cross-domain action transfer with practical applicability to films, games, and animation; code and weights are released.

Abstract

Action customization involves generating videos where the subject performs actions dictated by input control signals. Current methods use pose-guided or global motion customization but are limited by strict constraints on spatial structure, such as layout, skeleton, and viewpoint consistency, reducing adaptability across diverse subjects and scenarios. To overcome these limitations, we propose FlexiAct, which transfers actions from a reference video to an arbitrary target image. Unlike existing methods, FlexiAct allows for variations in layout, viewpoint, and skeletal structure between the subject of the reference video and the target image, while maintaining identity consistency. Achieving this requires precise action control, spatial structure adaptation, and consistency preservation. To this end, we introduce RefAdapter, a lightweight image-conditioned adapter that excels in spatial adaptation and consistency preservation, surpassing existing methods in balancing appearance consistency and structural flexibility. Additionally, based on our observations, the denoising process exhibits varying levels of attention to motion (low frequency) and appearance details (high frequency) at different timesteps. So we propose FAE (Frequency-aware Action Extraction), which, unlike existing methods that rely on separate spatial-temporal architectures, directly achieves action extraction during the denoising process. Experiments demonstrate that our method effectively transfers actions to subjects with diverse layouts, skeletons, and viewpoints. We release our code and model weights to support further research at https://shiyi-zh0408.github.io/projectpages/FlexiAct/

Figures (10)

• Figure 1: Attention maps between our frequency-aware embeddings and video tokens in the MMDiT at different denoising timesteps. Our embeddings focus on low-frequency motion information (e.g., motion regions) in early denoising stages and shift to high-frequency details in later stages.
• Figure 2: Overview of FlexiAct. (1) The upper part illustrates RefAdapter's training, which conditions arbitrary frames to enable transitions across diverse spatial structures. (2) The lower part outlines FAE's training and inference, where attention weights of video tokens to the frequency-aware embedding are dynamically adjusted based on timesteps, facilitating action extraction.
• Figure 3: Results of transferring "turning" action to the target image using the pose-based method and the animation version of the global motion method.
• Figure 4: Qualitative comparison of action transfer from reference video (Ref Video) to target images with varying spatial structures. Red boxes highlight regions where the appearance deviates from the target image. Our method demonstrates superior performance in maintaining appearance consistency with the target image and motion fidelity to the reference video compared to other approaches.
• Figure 5: Qualitative results of ablation study.We ablate Frequency-aware Action Extraction (FAE) and RefAdapter, comparing the action transfer results from reference videos (Ref Video) to different subjects. Ablating FAE reduces action accuracy, demonstrating its effectiveness in action extraction. Ablating RefAdapter degrades both appearance consistency and action precision, proving its capability in spatial structure adaptation for cross-subject action transfer.