Auto-Regressive Diffusion for Generating 3D Human-Object Interactions

TL;DR

This work tackles Text-to-HOI generation by learning a continuous HOI token space with a Contrastive VAE and generating long, coherent sequences via an Autoregressive Diffusion Model. The ARDM uses a Mamba-based context encoder and a lightweight MLP denoiser, enabling fast, context-aware token predictions conditioned on text and object cues. Key contributions include a physically aware latent space through cVAE, an autoregressive diffusion framework that mitigates error accumulation, and extensive experiments showing improved fidelity and speed on OMOMO and BEHAVE. The approach promises practical impact for animation, games, and robotics by delivering realistic, controllable HOI sequences efficiently.

Abstract

Text-driven Human-Object Interaction (Text-to-HOI) generation is an emerging field with applications in animation, video games, virtual reality, and robotics. A key challenge in HOI generation is maintaining interaction consistency in long sequences. Existing Text-to-Motion-based approaches, such as discrete motion tokenization, cannot be directly applied to HOI generation due to limited data in this domain and the complexity of the modality. To address the problem of interaction consistency in long sequences, we propose an autoregressive diffusion model (ARDHOI) that predicts the next continuous token. Specifically, we introduce a Contrastive Variational Autoencoder (cVAE) to learn a physically plausible space of continuous HOI tokens, thereby ensuring that generated human-object motions are realistic and natural. For generating sequences autoregressively, we develop a Mamba-based context encoder to capture and maintain consistent sequential actions. Additionally, we implement an MLP-based denoiser to generate the subsequent token conditioned on the encoded context. Our model has been evaluated on the OMOMO and BEHAVE datasets, where it outperforms existing state-of-the-art methods in terms of both performance and inference speed. This makes ARDHOI a robust and efficient solution for text-driven HOI tasks

Figures (5)

• Figure 1: Text guided Human-Object interaction motion sequences generated by our ARDHOI.
• Figure 2: ARDHOI. The left model is our Contrastive VAE (cVAE) which learns continuous HOI tokens in a contrastive manner. It is trained in phase 1 and frozen in phase 2. The right part is the ARDM, which generates HOI tokens in an autoregressive style using diffusion. ARDM is trained in phase 2.
• Figure 3: MLP Denoiser and autoregressive denoising
• Figure 4: Qualitative comparison with current models. MDM shows strong perturbations. InterDiff and HOI-Diff present a stick-to-hand motion pattern. And their global orientation is not well controlled which leads to penetration.
• Figure 5: Comparison of PCA plots for HOI positive and negative examples. The plot right is with triplet loss and the plot left is without triplet loss.