GA3CE: Unconstrained 3D Gaze Estimation with Gaze-Aware 3D Context Encoding

TL;DR

GA3CE tackles unconstrained 3D gaze estimation by explicitly modeling the spatial relationship between a subject and scene context. It represents the subject as 3D pose and scene objects as 3D points, then normalizes these into an egocentric space using an egocentric transformation and $D^3$ positional encoding before feeding them into a 3D gaze transformer. The approach yields substantial gains, achieving 13%–37% reductions in mean angular error on GFIE, CAD-120, and GAFA, and ablation studies confirm the value of incorporating 3D pose and object context along with GA3CE. By enabling gaze estimation even when close-up eye views are unavailable, GA3CE offers a practical path toward robust 3D gaze understanding in real-world settings.

Abstract

We propose a novel 3D gaze estimation approach that learns spatial relationships between the subject and objects in the scene, and outputs 3D gaze direction. Our method targets unconstrained settings, including cases where close-up views of the subject's eyes are unavailable, such as when the subject is distant or facing away. Previous approaches typically rely on either 2D appearance alone or incorporate limited spatial cues using depth maps in the non-learnable post-processing step. Estimating 3D gaze direction from 2D observations in these scenarios is challenging; variations in subject pose, scene layout, and gaze direction, combined with differing camera poses, yield diverse 2D appearances and 3D gaze directions even when targeting the same 3D scene. To address this issue, we propose GA3CE: Gaze-Aware 3D Context Encoding. Our method represents subject and scene using 3D poses and object positions, treating them as 3D context to learn spatial relationships in 3D space. Inspired by human vision, we align this context in an egocentric space, significantly reducing spatial complexity. Furthermore, we propose D$^3$ (direction-distance-decomposed) positional encoding to better capture the spatial relationship between 3D context and gaze direction in direction and distance space. Experiments demonstrate substantial improvements, reducing mean angle error by 13%-37% compared to leading baselines on benchmark datasets in single-frame settings.

Figures (18)

• Figure 1: (a) Our method estimates 3D gaze direction from an RGBD image and camera intrinsics. (b) Prior work hu2023gfie estimates gaze from 2D representations, incorporating 3D spatial cues only as a non-learnable post-processing step. (c) Since direct 3D gaze estimation from 2D representations is challenging, we use 3D pose and object positions as intermediate representations. (d) We introduce gaze-aware 3D context encoding (GA3CE), transforming the 3D context into a unified directional space. (d1) This space aligns with a egocentric (subject-centered) view, normalized relative to the head position and oriented to the view direction. (d2) This alignment enables the decomposition of a 3D point into direction and distance in egocentric space, with D$^3$ positional encoding capturing their correlations. (e) The transformer then learns spatial relationships between the subject, objects, and 3D gaze.
• Figure 2: Pipeline overview. PE = positional encoding; ECT = egocentric transformation. First, we extract 3D pose and object positions as 3D context representations. To reduce their variation and better capture spatial relationships between 3D context and gaze direction, we apply GA3CE: gaze-aware 3D context encoding. Using egocentric transformation, we convert 3D context to a egocentric space, and encoding them into a high-dimensional feature space for direction and distance with D$^3$ positional encoding. Finally, the 3D gaze transformer learns spatial relationships between the 3D context and gaze.
• Figure 3: Illustration of object position sampling using the segment-everything approach mobile_sam. This method comprehensively identifies object positions in the scene, shown as red points in (c). The colored point cloud is shown for visualization purposes only.
• Figure 4: Visualization of geometric normalization in 3D context through the egocentric transformation. (b) and (c) show 2D renderings of the colored point cloud after applying the egocentric transformation with different rotation normalizations. Note that, for intuitive visualization of the figure, we used the colored point cloud and its 2D rendering instead of the 3D pose and object positions. They are used solely for visualization purpose.
• Figure 5: 2D visualization of D$^3$ positional encoding. Each point on a grid shows a dot product between the encoded reference point $\mathbf{x}_\text{ref}=(20, 20)$ and an encoded point $\mathbf{x}\in\Omega$ on a 2D grid as an unnormalized similarity score. (a) shows the standard positional encoding tancik2020fourier, where similarity is high only at the reference point $\mathbf{x}_\text{ref}$ itself. (b) displays the proposed D$^3$ positional encoding, capturing both positional and directional similarities defined as $\tilde{\gamma}(\mathbf{x}_\text{ref}-\mathbf{c})^T\tilde{\gamma}(\mathbf{x}-\mathbf{c})$, given $\mathbf{c}=(6, 6)$ as the origin. This creates a radial gaze pattern from the origin $\mathbf{c}$ toward the reference point $\mathbf{x}_\text{ref}$, with similarity gradually increasing along the direction and distance from the origin, simulating a gaze effect.