Leveraging Multi-Modal Saliency and Fusion for Gaze Target Detection

TL;DR

This work tackles gaze target detection (GTD) in images by introducing a two-module framework that combines depth-aware saliency with multi-modal fusion. The Depth-infused Saliency Module (DISM) generates a depth-guided saliency map $S_i$ from $(D_i,M_i,F_i)$ using a 3D gaze-oriented projection, while the Multi-Modal Fusion (MMF) module blends scene, depth, and face cues with $S_i$ to produce a gaze heatmap $H_i$ trained via a regression objective. Across VideoAttentionTarget, GOO-Real, and GazeFollow, the method achieves state-of-the-art performance in AUC and competitive distance/angle metrics, with ablations confirming DISM and MMF contributions are critical for robustness in complex scenes. This approach advances GTD by integrating 3D scene understanding and multi-modal cues in an end-to-end trainable architecture, enabling more accurate gaze localization in real-world, in-frame settings.

Abstract

Gaze target detection (GTD) is the task of predicting where a person in an image is looking. This is a challenging task, as it requires the ability to understand the relationship between the person's head, body, and eyes, as well as the surrounding environment. In this paper, we propose a novel method for GTD that fuses multiple pieces of information extracted from an image. First, we project the 2D image into a 3D representation using monocular depth estimation. We then extract a depth-infused saliency module map, which highlights the most salient (\textit{attention-grabbing}) regions in image for the subject in consideration. We also extract face and depth modalities from the image, and finally fuse all the extracted modalities to identify the gaze target. We quantitatively evaluated our method, including the ablation analysis on three publicly available datasets, namely VideoAttentionTarget, GazeFollow and GOO-Real, and showed that it outperforms other state-of-the-art methods. This suggests that our method is a promising new approach for GTD.

Figures (6)

• Figure 1: Overview of our multi-modal saliency and fusion architecture for gaze target detection.
• Figure 2: Overview of DISM. We take 3D projection of depth map $P_d$ alongside gaze binning parameters $\theta_d$ and $\theta_{xy}$ to extract a sub-collection of filtered 3D points $P_c$. The re-projection of $P_c$ back to the image-plane serves as pseudo-labels for the FPN network, $f_{ds}$. The network provides a representation of the learned DISM map $S_i$.
• Figure 3: Our MMF module comprises three branches - face, scene, and depth. The three branches are fused in the Fusion module. The output of the module is a 2D Heatmap $H_i$ superimposed on the scene image $I_i$ here for visualization.
• Figure 4: Visualization results. This figure shows examples from the VideoAttentionTarget (first two rows), GazeFollow (middle two rows), and GOO-Real (last two rows) datasets. Each row shows the input image, depth map, DISM map, MMF heatmap, prediction result, and ground truth.
• Figure 5: Qualitative results. The red and green lines denote ground truth and predictions respectively. The first two rows represent the changes in gaze target points of a subject from video sequences in VideoAttentionTarget. The last two rows are images from GOO-Real for varying head poses.