MARRS: Masked Autoregressive Unit-based Reaction Synthesis

TL;DR

This work proposes MARRS, a novel framework designed to generate coordinated and fine-grained reaction motions using continuous representations and introduces Adaptive Unit Modulation (AUM) to facilitate interaction between body and hand units by using the information from one unit to adaptively modulate the other.

Abstract

This work aims at a challenging task: human action-reaction synthesis, i.e., generating human reactions conditioned on the action sequence of another person. Currently, autoregressive modeling approaches with vector quantization (VQ) have achieved remarkable performance in motion generation tasks. However, VQ has inherent disadvantages, including quantization information loss, low codebook utilization, etc. In addition, while dividing the body into separate units can be beneficial, the computational complexity needs to be considered. Also, the importance of mutual perception among units is often neglected. In this work, we propose MARRS, a novel framework designed to generate coordinated and fine-grained reaction motions using continuous representations. Initially, we present the Unit-distinguished Motion Variational AutoEncoder (UD-VAE), which segments the entire body into distinct body and hand units, encoding each independently. Subsequently, we propose Action-Conditioned Fusion (ACF), which involves randomly masking a subset of reactive tokens and extracting specific information about the body and hands from the active tokens. Furthermore, we introduce Adaptive Unit Modulation (AUM) to facilitate interaction between body and hand units by using the information from one unit to adaptively modulate the other. Finally, for the diffusion model, we employ a compact MLP as a noise predictor for each distinct body unit and incorporate the diffusion loss to model the probability distribution of each token. Both quantitative and qualitative results demonstrate that our method achieves superior performance. The code will be released upon acceptance.

Figures (6)

• Figure 1: Left: Paradigm comparison of different frameworks. (a) and (b) present the structures of the VQ-VAE-based and Diffusion-based methods, respectively, while (c) shows the framework of our proposed MARRS. Right: result comparison among our method and other methods on eight metrics.
• Figure 2: The overall framework of our proposed MARRS. (a) Whole-body motion is divided into two units: body and hands and then each unit is encoded independently by a VAE. (b) shows the process of the masked reaction generation model.
• Figure 3: Qualitative comparison with ReGenNet on human reaction synthesis. Blue for actors and Red for reactors.
• Figure 4: Visualization of inference process. The generation of entire tokens is performed in an autoregressive manner. Compact diffusion model is very small, consisting of only a 3-layer MLP. Therefore, MARRS can achieve fast inference speed.
• Figure 5: Qualitative comparison with ReGenNet on human reaction synthesis. Blue for actors and Red for reactors. The sequences generated by MARRS are more consistent with the action.