CoDance: An Unbind-Rebind Paradigm for Robust Multi-Subject Animation

TL;DR

CoDance tackles robust multi-subject character animation under pose misalignment by introducing an Unbind-Rebind paradigm that decouples motion semantics from spatial context and then rebinds motion to specific subjects using semantic text guidance and mask-based spatial constraints. The Unbind module perturbs pose inputs and latent features to learn location-agnostic motion representations, while the Rebind module uses a text branch and subject masks to reassign motion to intended subjects. It leverages a Diffusion Transformer backbone with VAE latent encoding and LoRA fine-tuning, and trains with mixed text-to-video data to strengthen semantic binding. Evaluations on CoDanceBench and Follow-Your-Pose-V2 show state-of-the-art performance and strong generalization to arbitrary subject counts and layouts, with open-source code planned.

Abstract

Character image animation is gaining significant importance across various domains, driven by the demand for robust and flexible multi-subject rendering. While existing methods excel in single-person animation, they struggle to handle arbitrary subject counts, diverse character types, and spatial misalignment between the reference image and the driving poses. We attribute these limitations to an overly rigid spatial binding that forces strict pixel-wise alignment between the pose and reference, and an inability to consistently rebind motion to intended subjects. To address these challenges, we propose CoDance, a novel Unbind-Rebind framework that enables the animation of arbitrary subject counts, types, and spatial configurations conditioned on a single, potentially misaligned pose sequence. Specifically, the Unbind module employs a novel pose shift encoder to break the rigid spatial binding between the pose and the reference by introducing stochastic perturbations to both poses and their latent features, thereby compelling the model to learn a location-agnostic motion representation. To ensure precise control and subject association, we then devise a Rebind module, leveraging semantic guidance from text prompts and spatial guidance from subject masks to direct the learned motion to intended characters. Furthermore, to facilitate comprehensive evaluation, we introduce a new multi-subject CoDanceBench. Extensive experiments on CoDanceBench and existing datasets show that CoDance achieves SOTA performance, exhibiting remarkable generalization across diverse subjects and spatial layouts. The code and weights will be open-sourced.

Figures (5)

• Figure 1: Multi-subject animations generated by CoDance. Given a single (potentially misaligned) driving pose sequence and one multi-subject reference image, CoDance produces coordinated, pose-controllable group dances without per-subject spatial alignment.
• Figure 2: The illustration of CoDance motivation. Although excelling at single-person animation, prior methods fail when handling multiple subjects due to a rigid binding between the reference and target pose, which results in mismatched outputs. By contrast, our Unbind-Rebind method successfully decouples motion from appearance, yielding compelling results.
• Figure 3: The pipeline of CoDance. Given a reference image $I^r$, a driving pose sequence $I^p_{1:F}$, a text prompt $\mathcal{T}$, and a subject mask $\mathcal{M}$, our model generates an animation video $I^g_{1:F}$. A VAE encoder extracts the latent feature $f^r_e$ from $I^r$. The Unbind module, implemented as a Pose Shift Encoder, processes $I^p_{1:F}$ to produce pose features. These are concatenated with patchified tokens from the noisy latent input for the DiT backbone. The Rebind module provides dual guidance: semantic features from a umT5 text encoder are injected via cross-attention, while spatial features from a Mask Encoder are added element-wise to the noisy latent. To bolster the model's semantic comprehension, the training process alternates between animation data (with probability $p_\text{ani}$) and a diverse text-to-video dataset (with probability $1-p_\text{ani}$). The DiT is initialized from a pretrained T2V model and fine-tuned using LoRA. Finally, a VAE decoder reconstructs the video. Note that the Unbind module and mixed-data training are applied exclusively during the training phase.
• Figure 4: Qualitative comparisons with SOTA methods.
• Figure 5: Qualitative results of ablation study.