Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Explainable Interfaces for Rapid Gaze-Based Interactions in Mixed Reality

Mengjie Yu, Dustin Harris, Ian Jones, Ting Zhang, Yue Liu, Naveen Sendhilnathan, Narine Kokhlikyan, Fulton Wang, Co Tran, Jordan L. Livingston, Krista E. Taylor, Zhenhong Hu, Mary A. Hood, Hrvoje Benko, Tanya R. Jonker

TL;DR

The paper investigates explainable AI for gaze-based interactions in mixed reality, addressing the opacity of high-accuracy black-box models. It introduces a real-time, multi-level XAI interface built around a Temporal Convolutional Network that predicts target selections from gaze data and SHAP counterfactual explanations to guide user behavior. In a between-subjects study with 32 participants, the XAI condition yielded higher selection accuracy ($F1$) and encouraged more nuanced gaze strategies, while user feedback highlighted preference for real-time, simple explanations that map to user behavior. The findings demonstrate that XAI can enhance human-model collaboration in XR and inform design guidelines for transparent, efficient gaze-based interfaces.

Abstract

Gaze-based interactions offer a potential way for users to naturally engage with mixed reality (XR) interfaces. Black-box machine learning models enabled higher accuracy for gaze-based interactions. However, due to the black-box nature of the model, users might not be able to understand and effectively adapt their gaze behaviour to achieve high quality interaction. We posit that explainable AI (XAI) techniques can facilitate understanding of and interaction with gaze-based model-driven system in XR. To study this, we built a real-time, multi-level XAI interface for gaze-based interaction using a deep learning model, and evaluated it during a visual search task in XR. A between-subjects study revealed that participants who interacted with XAI made more accurate selections compared to those who did not use the XAI system (i.e., F1 score increase of 10.8%). Additionally, participants who used the XAI system adapted their gaze behavior over time to make more effective selections. These findings suggest that XAI can potentially be used to assist users in more effective collaboration with model-driven interactions in XR.

Explainable Interfaces for Rapid Gaze-Based Interactions in Mixed Reality

TL;DR

The paper investigates explainable AI for gaze-based interactions in mixed reality, addressing the opacity of high-accuracy black-box models. It introduces a real-time, multi-level XAI interface built around a Temporal Convolutional Network that predicts target selections from gaze data and SHAP counterfactual explanations to guide user behavior. In a between-subjects study with 32 participants, the XAI condition yielded higher selection accuracy () and encouraged more nuanced gaze strategies, while user feedback highlighted preference for real-time, simple explanations that map to user behavior. The findings demonstrate that XAI can enhance human-model collaboration in XR and inform design guidelines for transparent, efficient gaze-based interfaces.

Abstract

Gaze-based interactions offer a potential way for users to naturally engage with mixed reality (XR) interfaces. Black-box machine learning models enabled higher accuracy for gaze-based interactions. However, due to the black-box nature of the model, users might not be able to understand and effectively adapt their gaze behaviour to achieve high quality interaction. We posit that explainable AI (XAI) techniques can facilitate understanding of and interaction with gaze-based model-driven system in XR. To study this, we built a real-time, multi-level XAI interface for gaze-based interaction using a deep learning model, and evaluated it during a visual search task in XR. A between-subjects study revealed that participants who interacted with XAI made more accurate selections compared to those who did not use the XAI system (i.e., F1 score increase of 10.8%). Additionally, participants who used the XAI system adapted their gaze behavior over time to make more effective selections. These findings suggest that XAI can potentially be used to assist users in more effective collaboration with model-driven interactions in XR.
Paper Structure (4 sections, 2 figures)

This paper contains 4 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Method
  3. Results
  4. Conclusion and Discussion

Figures (2)

  • Figure 1: Screenshots from the prototype showing the prediction navigator (a) and 3 Levels of Explanation (b)
  • Figure 2: Boxplot distribution of task performance and gaze behavior metrics between the XAI and control conditions. From left to right, F1 score, task completion time, gaze velocity and fixation duration, respectively.