Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GazeD: Context-Aware Diffusion for Accurate 3D Gaze Estimation

Riccardo Catalini, Davide Di Nucci, Guido Borghi, Davide Davoli, Lorenzo Garattoni, Giampiero Francesca, Yuki Kawana, Roberto Vezzani

TL;DR

GazeD tackles the ill-posed problem of 3D gaze estimation from a single RGB image by jointly predicting gaze and body pose using a diffusion model conditioned on 2D pose, surroundings, and scene context. The gaze is represented as an additional joint at a fixed distance from the eyes, enabling a unified pose-and-gaze regression framework and multi-hypothesis generation via diffusion with a DDIM scheduler. The method fuses near-body context through Deformable Context Extraction and far-scene context via a DETR-like object detector, producing multiple plausible gaze/pose hypotheses that can be aggregated with AVG or Oracle-based strategies. Empirical results on GAFA, GFIE, and Ego-Gaze show state-of-the-art 3D gaze accuracy, outperforming temporal methods while operating from a single RGB image, and also delivering competitive 3D pose estimates, highlighting the practical impact for real-world HCI, AR/VR, and robotics applications.

Abstract

We introduce GazeD, a new 3D gaze estimation method that jointly provides 3D gaze and human pose from a single RGB image. Leveraging the ability of diffusion models to deal with uncertainty, it generates multiple plausible 3D gaze and pose hypotheses based on the 2D context information extracted from the input image. Specifically, we condition the denoising process on the 2D pose, the surroundings of the subject, and the context of the scene. With GazeD we also introduce a novel way of representing the 3D gaze by positioning it as an additional body joint at a fixed distance from the eyes. The rationale is that the gaze is usually closely related to the pose, and thus it can benefit from being jointly denoised during the diffusion process. Evaluations across three benchmark datasets demonstrate that GazeD achieves state-of-the-art performance in 3D gaze estimation, even surpassing methods that rely on temporal information. Project details will be available at https://aimagelab.ing.unimore.it/go/gazed.

GazeD: Context-Aware Diffusion for Accurate 3D Gaze Estimation

TL;DR

GazeD tackles the ill-posed problem of 3D gaze estimation from a single RGB image by jointly predicting gaze and body pose using a diffusion model conditioned on 2D pose, surroundings, and scene context. The gaze is represented as an additional joint at a fixed distance from the eyes, enabling a unified pose-and-gaze regression framework and multi-hypothesis generation via diffusion with a DDIM scheduler. The method fuses near-body context through Deformable Context Extraction and far-scene context via a DETR-like object detector, producing multiple plausible gaze/pose hypotheses that can be aggregated with AVG or Oracle-based strategies. Empirical results on GAFA, GFIE, and Ego-Gaze show state-of-the-art 3D gaze accuracy, outperforming temporal methods while operating from a single RGB image, and also delivering competitive 3D pose estimates, highlighting the practical impact for real-world HCI, AR/VR, and robotics applications.

Abstract

We introduce GazeD, a new 3D gaze estimation method that jointly provides 3D gaze and human pose from a single RGB image. Leveraging the ability of diffusion models to deal with uncertainty, it generates multiple plausible 3D gaze and pose hypotheses based on the 2D context information extracted from the input image. Specifically, we condition the denoising process on the 2D pose, the surroundings of the subject, and the context of the scene. With GazeD we also introduce a novel way of representing the 3D gaze by positioning it as an additional body joint at a fixed distance from the eyes. The rationale is that the gaze is usually closely related to the pose, and thus it can benefit from being jointly denoised during the diffusion process. Evaluations across three benchmark datasets demonstrate that GazeD achieves state-of-the-art performance in 3D gaze estimation, even surpassing methods that rely on temporal information. Project details will be available at https://aimagelab.ing.unimore.it/go/gazed.
Paper Structure (17 sections, 1 equation, 6 figures, 7 tables)

This paper contains 17 sections, 1 equation, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. 3D Gaze Estimation
  4. 3D Human Pose Estimation
  5. Method
  6. Body & Surroundings
  7. Context with Objects
  8. Diffusion-based Multi-hypothesis Generation
  9. Experimental Evaluation
  10. Datasets
  11. Implementation Details and Training
  12. Baselines and Competitors
  13. Comparison with state-of-the-art
  14. Qualitative results
  15. Ablation Studies
  16. ...and 2 more sections

Figures (6)

  • Figure 1: GazeD method jointly predicts 3D gaze and body pose analyzing the 2D pose, the surroundings of the subject and the context, in terms of objects in the scene.
  • Figure 2: Human body skeleton with our additional gaze joint.
  • Figure 3: Overview of the proposed GazeD method that predicts the 3D gaze and human pose starting from a single input RGB image, combining information from the 2D body pose, surroundings, and context with objects.
  • Figure 4: Qualitative Results on the three datasets. Green arrow represents ground truth gaze direction, red arrow is the prediction
  • Figure 5: Ablation study on GFIE dataset: (a) the contribution of the "Context with objects" module in addition to the "Body&Surroundings". (b) Performance of GazeD in terms of MAE$_{\text{3D}}$ by varying the distance of the gaze joint from the head.
  • ...and 1 more figures