InterMask: 3D Human Interaction Generation via Collaborative Masked Modeling

TL;DR

InterMask introduces a two-stage framework for text-conditioned 3D human-human interaction generation that operates in discrete space. A shared VQ-VAE encodes each individual’s motion into a 2D token map, preserving spatial and temporal structure, and a dedicated Inter-M Transformer jointly models the two token streams using masked generative modeling with self, spatio-temporal, and cross-attention. The model is trained with a two-stage masking strategy and AdaLN-mod text conditioning, enabling high-fidelity, diverse interactions and out-of-the-box reaction generation without task-specific fine-tuning. Empirical results on InterHuman and InterX show state-of-the-art FID and strong text alignment, with efficient inference and robust performance across body representations; limitations include visual penetrations and biases toward dancing, suggesting directions for smoother transitions and longer-sequence modeling.

Abstract

Generating realistic 3D human-human interactions from textual descriptions remains a challenging task. Existing approaches, typically based on diffusion models, often produce results lacking realism and fidelity. In this work, we introduce InterMask, a novel framework for generating human interactions using collaborative masked modeling in discrete space. InterMask first employs a VQ-VAE to transform each motion sequence into a 2D discrete motion token map. Unlike traditional 1D VQ token maps, it better preserves fine-grained spatio-temporal details and promotes spatial awareness within each token. Building on this representation, InterMask utilizes a generative masked modeling framework to collaboratively model the tokens of two interacting individuals. This is achieved by employing a transformer architecture specifically designed to capture complex spatio-temporal inter-dependencies. During training, it randomly masks the motion tokens of both individuals and learns to predict them. For inference, starting from fully masked sequences, it progressively fills in the tokens for both individuals. With its enhanced motion representation, dedicated architecture, and effective learning strategy, InterMask achieves state-of-the-art results, producing high-fidelity and diverse human interactions. It outperforms previous methods, achieving an FID of $5.154$ (vs $5.535$ of in2IN) on the InterHuman dataset and $0.399$ (vs $5.207$ of InterGen) on the InterX dataset. Additionally, InterMask seamlessly supports reaction generation without the need for model redesign or fine-tuning.

Figures (16)

• Figure 1: InterMask generates high fidelity text-conditioned 3D human interactions, with accurate spatial and temporal coordination, including synchronized dance moves, realistic reaction timings in boxing, and correct proximity, while maintaining high-quality poses.
• Figure 2: Overview of InterMask. (a) Individual motions are quantized through vector quantization (VQ) to obtain 2D tokens $\{t_a, t_b\}$ for each. (b) Motion tokens from both individuals are flattened, concatenated, masked and predicted collaboratively by the Inter-M Transformer. (c) Each block in Inter-M Transformer consists of Self, Spatio-Temporal and Cross Attention modules to learn complex spatio-temporal dependencies within and between both interacting individuals.
• Figure 2: Ablation Study results on InterHuman test set to verify key components of the proposed Motion VQ-VAE. Bold face indicates the best result.
• Figure 3: Inference process. Starting from completely masked token sequences of both individuals $\{t_a(0), t_b(0)\}$, the Inter-M transformer generates all tokens in $I$ iterations. Next, the tokens are dequantized and decoded to generate motion sequences $\{\mathbf{m}_a, \mathbf{m}_b\}$ using the VQ-VAE decoder.
• Figure 4: Qualitative comparison between InterMask and InterGen liang2024intergen, highlighting InterMask's superior interaction quality, text adherence and avoidance of implicit biases.