HOIMotion: Forecasting Human Motion During Human-Object Interactions Using Egocentric 3D Object Bounding Boxes

TL;DR

HOIMotion tackles forecasting human motion during human–object interactions by integrating past body poses with egocentric 3D object bounding boxes. It proposes a graph-based encoder–residual–decoder pipeline: pose features are extracted via an encoder GCN and a pose residual GCN, object/head features are obtained with MLPs, and all are fused into a pose–object graph before forecasting with a residual decoder GCN. The approach yields consistent MPJPE improvements of up to 8.7% on ADT and 7.2% on MoGaze over pose-only baselines, with user studies indicating higher perceived precision and realism. The results underscore the significant information content carried by egocentric object bounding boxes for motion forecasting and point to practical benefits for AR/VR applications, including real-time, object-aware prediction capabilities.

Abstract

We present HOIMotion - a novel approach for human motion forecasting during human-object interactions that integrates information about past body poses and egocentric 3D object bounding boxes. Human motion forecasting is important in many augmented reality applications but most existing methods have only used past body poses to predict future motion. HOIMotion first uses an encoder-residual graph convolutional network (GCN) and multi-layer perceptrons to extract features from body poses and egocentric 3D object bounding boxes, respectively. Our method then fuses pose and object features into a novel pose-object graph and uses a residual-decoder GCN to forecast future body motion. We extensively evaluate our method on the Aria digital twin (ADT) and MoGaze datasets and show that HOIMotion consistently outperforms state-of-the-art methods by a large margin of up to 8.7% on ADT and 7.2% on MoGaze in terms of mean per joint position error. Complementing these evaluations, we report a human study (N=20) that shows that the improvements achieved by our method result in forecasted poses being perceived as both more precise and more realistic than those of existing methods. Taken together, these results reveal the significant information content available in egocentric 3D object bounding boxes for human motion forecasting and the effectiveness of our method in exploiting this information.

Figures (3)

• Figure 1: Our method for egocentric scene object-aware human motion forecasting uses an encoder GCN and a pose residual GCN to extract features from historical body poses and employs three MLPs to respectively extract features from head orientations, and the bounding boxes of static and dynamic scene objects in the egocentric view. The pose, head, and object features are fused into a novel pose-object graph and a fuse residual GCN and a decoder GCN are applied to forecast future body motion from the pose-object graph.
• Figure 2: Visualisation of the prediction results from different methods on a sample of the ADT dataset pan2023aria. Our method can accurately predict the future body motion of squat down to touch an object while prior methods that only use historical body poses fail to predict this motion.
• Figure 3: Visualisation of different ablated versions of our method on a sample of the MoGaze dataset kratzer2020mogaze. Our method consistently outperforms the ablated versions at different future time horizons.