Int3DNet: Scene-Motion Cross Attention Network for 3D Intention Prediction in Mixed Reality

Abstract

We propose Int3DNet, a scene-aware network that predicts 3D intention areas directly from scene geometry and head-hand motion cues, enabling robust human intention prediction without explicit object-level perception. In Mixed Reality (MR), intention prediction is critical as it enables the system to anticipate user actions and respond proactively, reducing interaction delays and ensuring seamless user experiences. Our method employs a cross attention fusion of sparse motion cues and scene point clouds, offering a novel approach that directly interprets the user's spatial intention within the scene. We evaluated Int3DNet on MoGaze and CIRCLE datasets, which are public datasets for full-body human-scene interactions, showing consistent performance across time horizons of up to 1500 ms and outperforming the baselines, even in diverse and unseen scenes. Moreover, we demonstrate the usability of proposed method through a demonstration of efficient visual question answering (VQA) based on intention areas. Int3DNet provides reliable 3D intention areas derived from head-hand motion and scene geometry, thus enabling seamless interaction between humans and MR systems through proactive processing of intention areas.

Figures (6)

• Figure 1: Architecture of Int3DNet. Scene point clouds, past head–hands trajectories, and head orientations are encoded via PointNet++, ST-GCN, and DCT–MLP modules. Motion and orientation features are fused with scene features through a cross attention, and the output head predicts the future intention area. The network is trained with a combination of weighted BCE, focal, and dice losses.
• Figure 2: Examples from two datasets. The CIRCLE dataset (left column) contains complex scene configurations, while the MoGaze dataset (right column) contains simpler scenes. Reprinted from araujo2023circle and kratzer2020mogaze.
• Figure 3: Qualitative results of intention prediction. The columns show, from left to right, the ground truth, Head Orientation model, Motion Forecast model, and our method. Rows 1–3 correspond to the CIRCLE dataset, and rows 4–6 to the MoGaze dataset.
• Figure 4: Pipeline of the intention-based VQA application. Past RGB-D, device, and hand trajectories are used to reconstruct the scene and intention prediction. Int3DNet predicts the future intention area, which is projected to 2D image plane and fed into a VLM for reasoning about the intention area.
• Figure 5: Qualitative results of the intention-based VQA. The right panel shows candidate objects: gray text indicates full-view image outputs, while bold text shows extracted intention area outputs. Blue boxes highlight ground truth target objects. Using the intention area reduces the candidate set for more precise reasoning.