in2IN: Leveraging individual Information to Generate Human INteractions

TL;DR

The paper tackles text-driven human–human motion generation by introducing in2IN, a diffusion model conditioned on both global interaction descriptions and granular individual descriptions, improved through multi-weight classifier-free guidance. It extends the InterHuman dataset with LLM-generated individual motions and achieves state-of-the-art results on InterHuman, while also proposing DualMDM, a model-composition approach that blends the interaction model with a single-person motion prior to enrich intra-personal diversity. The combination of in2IN and DualMDM provides finer control over both inter- and intra-personal dynamics, enabling more realistic and varied human interactions conditioned on text. This work advances practical capabilities for robotics, gaming, and animation where nuanced, coherent multi-person motions are required, and it introduces metrics like Extrinsic Individual Diversity to quantify intra-personal control. Key technical contributions include (i) a Siamese Transformer diffusion architecture for dual-condition generation, (ii) a multi-weight CFG scheme for independent conditioning on interaction and individual descriptions, and (iii) a variable-weight DualMDM scheduler that blends diffusion outputs over the denoising steps to maximize intra-person diversity without sacrificing inter-person coherence.

Abstract

Generating human-human motion interactions conditioned on textual descriptions is a very useful application in many areas such as robotics, gaming, animation, and the metaverse. Alongside this utility also comes a great difficulty in modeling the highly dimensional inter-personal dynamics. In addition, properly capturing the intra-personal diversity of interactions has a lot of challenges. Current methods generate interactions with limited diversity of intra-person dynamics due to the limitations of the available datasets and conditioning strategies. For this, we introduce in2IN, a novel diffusion model for human-human motion generation which is conditioned not only on the textual description of the overall interaction but also on the individual descriptions of the actions performed by each person involved in the interaction. To train this model, we use a large language model to extend the InterHuman dataset with individual descriptions. As a result, in2IN achieves state-of-the-art performance in the InterHuman dataset. Furthermore, in order to increase the intra-personal diversity on the existing interaction datasets, we propose DualMDM, a model composition technique that combines the motions generated with in2IN and the motions generated by a single-person motion prior pre-trained on HumanML3D. As a result, DualMDM generates motions with higher individual diversity and improves control over the intra-person dynamics while maintaining inter-personal coherence.

Figures (11)

• Figure 1: We present in2IN, a diffusion model architecture capable of generating human-human motion interactions using general interaction descriptions to model the inter-personal dynamics and specific individual descriptions to model the intra-personal dynamics. Furthermore, we propose DualMDM, a motion composition method that is able to combine predictions made by an interaction model and by a single-person motion prior, thus increasing the intra-personal diversity of human motion interactions.
• Figure 2: in2IN diffusion model. Our proposed architecture consists of a Siamese Transformer that generates the denoised motion of each individual in the interaction ($x^0_a$ and $x^0_b$). First, a self-attention layer models the intra-personal dependencies using the encoded individual condition and noisy motion of each person ($x^t_a$ and $x^t_b$). Then, a cross-attention module models the inter-personal dynamics using the encoded interaction description, the self-attention output, and the noisy motion from the other interacting person.
• Figure 3: Different weights schedulers tested for DualMDM: Exponential , Inverse Exponential, Constant, and Linear.
• Figure 4: R-Precision and FID for the different weights on the Multi-Weight CFG tested in isolation. Each column ablates a different weight ($w_c$, $w_I$, $w_i$). $w_c$ has been ablated with $w_I{=}w_i{=}0$. $w_I$ and $w_i$ with $w_c{=}1$, and the other weight set to 0.
• Figure 5: Interaction Description: The two guys meet, grip each other's hand, and nod in agreement. The X-axis represents time.