VQ-Style: Disentangling Style and Content in Motion with Residual Quantized Representations

TL;DR

This work tackles the challenge of disentangling style from content in human motion by introducing a Residual Vector Quantized VAE (RVQ-VAE) that learns a coarse-to-fine motion representation. Content is captured in early codebooks while style is stored in later codebooks, with training-time contrastive and mutual information losses to prevent leakage and improve separation. Inference-time style transfer is achieved via Quantized Code Swapping, swapping content and style codes across residual layers to produce stylized motions without fine-tuning on unseen styles. The approach enables robust style transfer, style transitions, inversion, blending, and data augmentation, achieving strong zero-shot generalization and competitive content preservation across multiple motion datasets.

Abstract

Human motion data is inherently rich and complex, containing both semantic content and subtle stylistic features that are challenging to model. We propose a novel method for effective disentanglement of the style and content in human motion data to facilitate style transfer. Our approach is guided by the insight that content corresponds to coarse motion attributes while style captures the finer, expressive details. To model this hierarchy, we employ Residual Vector Quantized Variational Autoencoders (RVQ-VAEs) to learn a coarse-to-fine representation of motion. We further enhance the disentanglement by integrating contrastive learning and a novel information leakage loss with codebook learning to organize the content and the style across different codebooks. We harness this disentangled representation using our simple and effective inference-time technique Quantized Code Swapping, which enables motion style transfer without requiring any fine-tuning for unseen styles. Our framework demonstrates strong versatility across multiple inference applications, including style transfer, style removal, and motion blending.

Figures (14)

• Figure 1: Our approach encodes motion into a number of codebooks stacked in a residual manner. Our training strategy enables the content to be represented by the first (blue) codebook, while style is represented by the downstream codebooks in red.
• Figure 2: Quantized Latent Code Swapping. To transfer style from one clip onto another, we encode both clips, and decode a combined embedding constructed by adding the content code (shown in blue) from the content clip and the style code (shown in red) from the second clip.
• Figure 3: TSNE plots of residual embeddings illustrating style-content disentanglement in latent space. The colors indicated different style labels. (a) Without contrastive learning, styles begin to weakly cluster from the second codebook onward. Embedding 1 remains unclustered, as intended. (b) With contrastive learning, the separation between styles becomes more pronounced. (c) With mutual information loss the disentanglement is further enhanced. (d) The model also disentangles styles never seen during training, demonstrating its generalization ability.
• Figure 4: Content extraction by decoding motion using only the first codebook. Left (blue): ground truth, Middle (pink): full reconstruction, Right (texture): extracted content. Best viewed in color.
• Figure 5: Interpolation between content and stylized motion. Going from left to right, more of the style code is removed. The resulting motion smoothly approaches Neutral motion.