Novel Diffusion Models for Multimodal 3D Hand Trajectory Prediction

TL;DR

This work introduces MMTwin, a diffusion-based framework for 3D hand trajectory prediction in egocentric settings that jointly models future headset egomotion and hand motions using twin diffusion models. By integrating multimodal inputs—2D RGB images, 3D point clouds, past hand waypoints, and text prompts—and employing a hybrid Mamba-Transformer denoising module, MMTwin achieves superior 3D HTP performance and strong generalization to unseen environments. The approach decouples egomotion and hand motion while leveraging their synergy through conditioned denoising, and demonstrates state-of-the-art results across multiple datasets, including real-world data collected with a low-cost headset. These results highlight the practical potential for improved robot manipulation and human-robot interaction in multimodal, real-time settings.

Abstract

Predicting hand motion is critical for understanding human intentions and bridging the action space between human movements and robot manipulations. Existing hand trajectory prediction (HTP) methods forecast the future hand waypoints in 3D space conditioned on past egocentric observations. However, such models are only designed to accommodate 2D egocentric video inputs. There is a lack of awareness of multimodal environmental information from both 2D and 3D observations, hindering the further improvement of 3D HTP performance. In addition, these models overlook the synergy between hand movements and headset camera egomotion, either predicting hand trajectories in isolation or encoding egomotion only from past frames. To address these limitations, we propose novel diffusion models (MMTwin) for multimodal 3D hand trajectory prediction. MMTwin is designed to absorb multimodal information as input encompassing 2D RGB images, 3D point clouds, past hand waypoints, and text prompt. Besides, two latent diffusion models, the egomotion diffusion and the HTP diffusion as twins, are integrated into MMTwin to predict camera egomotion and future hand trajectories concurrently. We propose a novel hybrid Mamba-Transformer module as the denoising model of the HTP diffusion to better fuse multimodal features. The experimental results on three publicly available datasets and our self-recorded data demonstrate that our proposed MMTwin can predict plausible future 3D hand trajectories compared to the state-of-the-art baselines, and generalizes well to unseen environments. The code and pretrained models have been released at https://github.com/IRMVLab/MMTwin.

Figures (7)

• Figure I: MMTwin receives multimodal data to concurrently predict future camera egomotion and hand trajectories with twin diffusion models. It attends to 3D structure awareness and synergy between hand movements and camera egomotion in future time periods.
• Figure II: Our proposed MMTwin (a) extracts features from multimodal data, and (b) decouples predictions of future camera egomotion features and 3D hand trajectories by novel twin diffusion models. The vanilla Mamba (VM) is used for denoising in the egomotion diffusion. We further design a new denoising model in HTP diffusion with (c) a hybrid Mamba-Transformer module (HMTM), encompassing the egomotion-aware Mamba (EAM) blocks and (d) the structure-aware Transformer (SAT).
• Figure III: Projection-based hand removal. We use MobileSAM mobile_sam to generate the hand mask for each input image, and filter out the 3D points that are projected into the hand area by camera intrinsics.
• Figure IV: The exampled head movement (corresponding to camera egomotion) and hand movement coupled during the hand-object interaction process in egocentric views in the EgoPAT3D dataset li2022egocentric.
• Figure V: We use a headset RGBD camera (a) to obtain self-recorded data (b). Here we also visualize the corresponding MMTwin predictions after 10 denoising processes, projected to the image plane (c), where MMTwin predictions and ground-truth future hand waypoints are represented as red and green points respectively.