Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Human Visual Attention on 3D Surfaces through Geometry-Queried Semantic Priors

Soham Pahari, Sandeep C. Kumain

TL;DR

This work tackles the challenge of modeling human visual attention on 3D surfaces by integrating bottom-up geometric cues with top-down semantic priors. It introduces SemGeo-AttentionNet, a dual-stream architecture where geometry queries diffusion-based semantic content through asymmetric cross-attention, and extends static saliency to mesh-aware scanpaths via reinforcement learning on surface topology. Key contributions include diffusion-distilled semantic priors, geometry-to-semantics fusion, and a mesh-respecting RL framework for scanpaths, with state-of-the-art results on SAL3D, NUS3D, and 3DVA across distributional and structural metrics. The approach offers a cognitively grounded 3D attention model with practical implications for VR/AR gaze analytics and 3D scene understanding.

Abstract

Human visual attention on three-dimensional objects emerges from the interplay between bottom-up geometric processing and top-down semantic recognition. Existing 3D saliency methods rely on hand-crafted geometric features or learning-based approaches that lack semantic awareness, failing to explain why humans fixate on semantically meaningful but geometrically unremarkable regions. We introduce SemGeo-AttentionNet, a dual-stream architecture that explicitly formalizes this dichotomy through asymmetric cross-modal fusion, leveraging diffusion-based semantic priors from geometry-conditioned multi-view rendering and point cloud transformers for geometric processing. Cross-attention ensures geometric features query semantic content, enabling bottom-up distinctiveness to guide top-down retrieval. We extend our framework to temporal scanpath generation through reinforcement learning, introducing the first formulation respecting 3D mesh topology with inhibition-of-return dynamics. Evaluation on SAL3D, NUS3D and 3DVA datasets demonstrates substantial improvements, validating how cognitively motivated architectures effectively model human visual attention on three-dimensional surfaces.

Learning Human Visual Attention on 3D Surfaces through Geometry-Queried Semantic Priors

TL;DR

This work tackles the challenge of modeling human visual attention on 3D surfaces by integrating bottom-up geometric cues with top-down semantic priors. It introduces SemGeo-AttentionNet, a dual-stream architecture where geometry queries diffusion-based semantic content through asymmetric cross-attention, and extends static saliency to mesh-aware scanpaths via reinforcement learning on surface topology. Key contributions include diffusion-distilled semantic priors, geometry-to-semantics fusion, and a mesh-respecting RL framework for scanpaths, with state-of-the-art results on SAL3D, NUS3D, and 3DVA across distributional and structural metrics. The approach offers a cognitively grounded 3D attention model with practical implications for VR/AR gaze analytics and 3D scene understanding.

Abstract

Human visual attention on three-dimensional objects emerges from the interplay between bottom-up geometric processing and top-down semantic recognition. Existing 3D saliency methods rely on hand-crafted geometric features or learning-based approaches that lack semantic awareness, failing to explain why humans fixate on semantically meaningful but geometrically unremarkable regions. We introduce SemGeo-AttentionNet, a dual-stream architecture that explicitly formalizes this dichotomy through asymmetric cross-modal fusion, leveraging diffusion-based semantic priors from geometry-conditioned multi-view rendering and point cloud transformers for geometric processing. Cross-attention ensures geometric features query semantic content, enabling bottom-up distinctiveness to guide top-down retrieval. We extend our framework to temporal scanpath generation through reinforcement learning, introducing the first formulation respecting 3D mesh topology with inhibition-of-return dynamics. Evaluation on SAL3D, NUS3D and 3DVA datasets demonstrates substantial improvements, validating how cognitively motivated architectures effectively model human visual attention on three-dimensional surfaces.
Paper Structure (53 sections, 12 equations, 4 figures, 11 tables)

This paper contains 53 sections, 12 equations, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Hand-crafted Geometric Methods
  4. Learning-based Approaches
  5. Texture and Semantic Integration
  6. Temporal Attention and Scanpath Modeling
  7. Methodology
  8. Overview and Attention Hypothesis
  9. Semantic Prior via Diffusion Feature Distillation
  10. Design rationale:
  11. Geometric Encoding and Efficient Sampling
  12. Asymmetric Geometry-to-Semantics Fusion
  13. Why this asymmetry matters:
  14. Training with Hybrid Loss
  15. Scanpath Generation via Reinforcement Learning
  16. ...and 38 more sections

Figures (4)

  • Figure 1: Method Overview. We render the input mesh from multiple viewpoints and extract semantic features from geometry-conditioned Stable Diffusion U-Net (1280-dim) and DINOv2 (768-dim). These features are unprojected to mesh vertices, yielding 2048-dim semantic descriptors that are fused with Point Transformer V3 geometric features through cross-attention to predict per-vertex saliency.
  • Figure 2: Results Gallery. SemGeoAttentionNet's performance on Sal3D dataset. Warm areas are the salient part.
  • Figure 3: Results Gallery. SemGeoAttentionNet's performance on NUS3D dataset. Warm represents the salient part and the comparision between GT and Prediction shows the accurateness of our model.
  • Figure 4: Comparison.. Our model's performance on Sal3D along side benchmark model martin2024sal3d and song2019mesh