HuBar: A Visual Analytics Tool to Explore Human Behaviour based on fNIRS in AR guidance systems

TL;DR

HuBar addresses the challenge of understanding human behavior during non-linear AR-guided tasks by combining multimodal time-series data (fNIRS, gaze, IMU, and video) within a four-view visual analytics interface. The system enables cross-session comparisons and detailed inspection through linked views (Overview, Event Timeline, Summary Matrix, Detail View) and is validated on the Ocarina dataset of helicopter copilot training with expert interviews and a high SUS score. Through two case studies, HuBar demonstrates its ability to reveal correlations between cognitive workload, errors, and task procedures, aiding AR developers in refining guidance strategies. The work highlights practical implications for post-hoc analysis and coaching in safety-critical AR applications, while outlining limitations and directions for real-time monitoring and broader deployment.

Abstract

The concept of an intelligent augmented reality (AR) assistant has significant, wide-ranging applications, with potential uses in medicine, military, and mechanics domains. Such an assistant must be able to perceive the environment and actions, reason about the environment state in relation to a given task, and seamlessly interact with the task performer. These interactions typically involve an AR headset equipped with sensors which capture video, audio, and haptic feedback. Previous works have sought to facilitate the development of intelligent AR assistants by visualizing these sensor data streams in conjunction with the assistant's perception and reasoning model outputs. However, existing visual analytics systems do not focus on user modeling or include biometric data, and are only capable of visualizing a single task session for a single performer at a time. Moreover, they typically assume a task involves linear progression from one step to the next. We propose a visual analytics system that allows users to compare performance during multiple task sessions, focusing on non-linear tasks where different step sequences can lead to success. In particular, we design visualizations for understanding user behavior through functional near-infrared spectroscopy (fNIRS) data as a proxy for perception, attention, and memory as well as corresponding motion data (acceleration, angular velocity, and gaze). We distill these insights into embedding representations that allow users to easily select groups of sessions with similar behaviors. We provide two case studies that demonstrate how to use these visualizations to gain insights about task performance using data collected during helicopter copilot training tasks. Finally, we evaluate our approach by conducting an in-depth examination of a think-aloud experiment with five domain experts.

Figures (5)

• Figure 1: Uncovering Data Quality Issues. On the right side, the scatterplot showcases sessions clustered by their IMU data, with glyphs encoding the subject ID. Variations in session counts per subject are evident, with some outliers identified in the upper left corner and highlighted through lasso selection. On the left side, the event timeline view reveals missing data points in trials 8, 19, and 20, likely attributed to mission log failures. Despite an initially comprehensive appearance, Trial 20 exhibited notable gaps in fNIRS data.
• Figure 2: Workload Aggregation View. On the left side, a selection of sessions is made within the scatterplot to identify instances where the underloaded mental state prevails. On the right side, sessions are organized by trials, depicting workload and error contribution associated with each mental state across all categories for every trial group. Notably, Trial 13 reveals a substantial correlation between errors and underload state, as indicated by the prominent pink marker near the value of 1.
• Figure 3: Event Timeline and Detail Views. On the left side, the Event Timeline View presents sessions belonging to Trial 13, conducted by different subjects. Subject 293 demonstrates sustained optimal attention and minimal errors, while Subject 9636 encounters numerous errors under an underload attention state. On the right side, the Detail View displays IMU data for Subjects 293 and 9636, revealing distinct patterns in linear acceleration. Subject 293 exhibits consistent, controlled motion, while Subject 9636 shows considerable variation, indicative of frequent stops and starts.
• Figure 4: The Event Timeline View displays sessions conducted by Subject 4352 alongside error and workload summaries. The workload summary reveals a consistent overload mental state across all sessions, notably correlating with errors, particularly in Trials 2 and 4, regardless of task variations.
• Figure 5: Performance Overview for Subject 9636. The Timeline view, Matrix view, and Workload Summaries illustrate performance across three consecutive trials (same task conditions). Notable trends include consistent task execution, reduced errors (especially in Procedure E), increased errors in Procedure F that are correlated to the transition from the preflight (PF) to flight (FL) phase, and correlations between errors and mental states, particularly in Procedure E during the first trial where the overload mental state is correlated with errors, as shown in the tooltip. Workload summaries indicate mental state improvements, with the last trial showing predominantly optimal states.