Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GazeMoDiff: Gaze-guided Diffusion Model for Stochastic Human Motion Prediction

Haodong Yan, Zhiming Hu, Syn Schmitt, Andreas Bulling

TL;DR

GazeMoDiff is presented - a novel gaze-guided denoising diffusion model to generate stochastic human motions that outperforms the state-of-the-art methods by a large margin in terms of multi-modal final displacement error.

Abstract

Human motion prediction is important for many virtual and augmented reality (VR/AR) applications such as collision avoidance and realistic avatar generation. Existing methods have synthesised body motion only from observed past motion, despite the fact that human eye gaze is known to correlate strongly with body movements and is readily available in recent VR/AR headsets. We present GazeMoDiff - a novel gaze-guided denoising diffusion model to generate stochastic human motions. Our method first uses a gaze encoder and a motion encoder to extract the gaze and motion features respectively, then employs a graph attention network to fuse these features, and finally injects the gaze-motion features into a noise prediction network via a cross-attention mechanism to progressively generate multiple reasonable human motions in the future. Extensive experiments on the MoGaze and GIMO datasets demonstrate that our method outperforms the state-of-the-art methods by a large margin in terms of multi-modal final displacement error (17.3% on MoGaze and 13.3% on GIMO). We further conducted a human study (N=21) and validated that the motions generated by our method were perceived as both more precise and more realistic than those of prior methods. Taken together, these results reveal the significant information content available in eye gaze for stochastic human motion prediction as well as the effectiveness of our method in exploiting this information.

GazeMoDiff: Gaze-guided Diffusion Model for Stochastic Human Motion Prediction

TL;DR

GazeMoDiff is presented - a novel gaze-guided denoising diffusion model to generate stochastic human motions that outperforms the state-of-the-art methods by a large margin in terms of multi-modal final displacement error.

Abstract

Human motion prediction is important for many virtual and augmented reality (VR/AR) applications such as collision avoidance and realistic avatar generation. Existing methods have synthesised body motion only from observed past motion, despite the fact that human eye gaze is known to correlate strongly with body movements and is readily available in recent VR/AR headsets. We present GazeMoDiff - a novel gaze-guided denoising diffusion model to generate stochastic human motions. Our method first uses a gaze encoder and a motion encoder to extract the gaze and motion features respectively, then employs a graph attention network to fuse these features, and finally injects the gaze-motion features into a noise prediction network via a cross-attention mechanism to progressively generate multiple reasonable human motions in the future. Extensive experiments on the MoGaze and GIMO datasets demonstrate that our method outperforms the state-of-the-art methods by a large margin in terms of multi-modal final displacement error (17.3% on MoGaze and 13.3% on GIMO). We further conducted a human study (N=21) and validated that the motions generated by our method were perceived as both more precise and more realistic than those of prior methods. Taken together, these results reveal the significant information content available in eye gaze for stochastic human motion prediction as well as the effectiveness of our method in exploiting this information.
Paper Structure (26 sections, 7 equations, 4 figures, 4 tables)

This paper contains 26 sections, 7 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Human Motion Prediction
  4. Correlations between Eye Gaze and Body Motion
  5. Denoising Diffusion Models
  6. GazeMoDiff Model
  7. Gaze-motion Feature Extraction
  8. Gaze Encoder
  9. Motion Encoder
  10. Gaze-motion Feature Fusion
  11. Diffusion-based Motion Generation
  12. Training
  13. Experiments
  14. Datasets
  15. Evaluation Metrics
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Diffusion process and reversed process in DDPM.
  • Figure 2: Overview of the proposed method GazeMoDiff. GazeMoDiff first uses a gaze encoder and a motion encoder to extract the gaze and motion features respectively, then employs a spatio-temporal graph attention network to fuse these features, and finally injects the gaze-motion features into a noise prediction network via a cross-attention mechanism to generate multiple reasonable human future motions through a progressive denoising process.
  • Figure 3: Ground truth (GT) human pose at future one second and multiple pose predictions generated by different methods on the MoGaze dataset kratzer2020mogaze with the best prediction (lowest $l_2$ distance to GT) boxed in green. Our method can generate motions that are closer to the ground truth than the state-of-the-art method HumanMAC chen2023humanmac.
  • Figure 4: Ground truth (GT) human pose at future one second and multiple pose predictions generated by different methods on the MoGaze dataset kratzer2020mogaze with the best prediction (lowest $l_2$ distance to GT) boxed in green. Our method using eye gaze can generate motions that are closer to the ground truth than the ablated version of not using eye gaze.