MoCLIP: Motion-Aware Fine-Tuning and Distillation of CLIP for Human Motion Generation

TL;DR

MoCLIP is introduced, a fine-tuned CLIP model with an additional motion encoding head, trained on motion sequences using contrastive learning and tethering loss, establishing it as a robust framework for enhancing motion generation.

Abstract

Human motion generation is essential for fields such as animation, robotics, and virtual reality, requiring models that effectively capture motion dynamics from text descriptions. Existing approaches often rely on Contrastive Language-Image Pretraining (CLIP)-based text encoders, but their training on text-image pairs constrains their ability to understand temporal and kinematic structures inherent in motion and motion generation. This work introduces MoCLIP, a fine-tuned CLIP model with an additional motion encoding head, trained on motion sequences using contrastive learning and tethering loss. By explicitly incorporating motion-aware representations, MoCLIP enhances motion fidelity while remaining compatible with existing CLIP-based pipelines and seamlessly integrating into various CLIP-based methods. Experiments demonstrate that MoCLIP improves Top-1, Top-2, and Top-3 accuracy while maintaining competitive FID, leading to improved text-to-motion alignment results. These results highlight MoCLIP's versatility and effectiveness, establishing it as a robust framework for enhancing motion generation.

Figures (3)

• Figure 1: Overview of our MoCLIP training framework, which fine-tunes CLIP for human motion representation. We adopt M2T-Interpretable radouane2024guided as the motion encoder to extract spatio-temporal embeddings from a motion sequence. This encoder includes cross-limb attention to capture fine-grained inter-limb coordination. The resulting motion embeddings $M = \{M_0, M_1, ..., M_B\}$ are aligned with text embeddings via a contrastive loss. To preserve CLIP's broad semantic knowledge, we introduce a distillation loss (Tethering Loss), which constrains the student text encoder using the pre-trained teacher text encoder. The parameter $\lambda$ controls the influence of this constraint, balancing adaptation and semantic retention.
• Figure 2: Example integration of the MoCLIP model into existing text-to-motion generation pipelines. MoCLIP serves as a direct replacement for the standard CLIP encoder previously utilized in various established models, including MoMask guo2024momask, BAD hosseyni2024bad, and BAMM pinyoanuntapong2024bamm. The figure illustrates simplified architectures of the downstream models, demonstrating how MoCLIP substitutes standard CLIP providing more motion oriented text embeddings. MoCLIP is integrated by replacing the weights of the original CLIP model with MoCLIP then freezing for the training of the downstream models.
• Figure 3: Ablation studies examining the impact of fine-tuning duration in naive training (top row) and tethering loss weight $\lambda$ (bottom row) on model performance, measured by Frechet Inception Distance (FID, green axis) and Multimodal Distance (MM-Dist, blue axis). Each plot compares different fine-tuning epochs (2, 5, 7, and 10 epochs) for naive baseline training (top) and varying tethering strengths ($\lambda$ from 0.0 to 1.0) for advanced MoCLIP model and training (bottom). Lower values indicate improved perceptual quality (FID) and better multimodal alignment (MM-Dist). Results are averaged over multiple runs on the HumanML3D dataset.