Explainable XR: Understanding User Behaviors of XR Environments using LLM-assisted Analytics Framework

TL;DR

EXplainable XR (EXR) tackles fragmentation in XR user analytics across AR, VR, and MR by introducing a unified framework centered on a standardized User Action Descriptor (UAD), a Unity-based session recorder, and a web-based visual analytics interface augmented with LLM-assisted insights. The approach enables cross-virtuality, multi-user collaboration, and multimodal data analysis, enhanced by a multi-agent LLM analytics pipeline that generates explainable insights and guided exploration through Analytics Insight and AoI Markers. Five XR use-case demonstrations and a user study show high usability and the ability to extract multifaceted, actionable patterns and intentions from immersive sessions. The work highlights the value of action-centric logging and distributed AI-assisted reasoning for interpretable XR analytics, while outlining privacy, real-time, and robustness enhancements for future development.

Abstract

We present Explainable XR, an end-to-end framework for analyzing user behavior in diverse eXtended Reality (XR) environments by leveraging Large Language Models (LLMs) for data interpretation assistance. Existing XR user analytics frameworks face challenges in handling cross-virtuality - AR, VR, MR - transitions, multi-user collaborative application scenarios, and the complexity of multimodal data. Explainable XR addresses these challenges by providing a virtuality-agnostic solution for the collection, analysis, and visualization of immersive sessions. We propose three main components in our framework: (1) A novel user data recording schema, called User Action Descriptor (UAD), that can capture the users' multimodal actions, along with their intents and the contexts; (2) a platform-agnostic XR session recorder, and (3) a visual analytics interface that offers LLM-assisted insights tailored to the analysts' perspectives, facilitating the exploration and analysis of the recorded XR session data. We demonstrate the versatility of Explainable XR by demonstrating five use-case scenarios, in both individual and collaborative XR applications across virtualities. Our technical evaluation and user studies show that Explainable XR provides a highly usable analytics solution for understanding user actions and delivering multifaceted, actionable insights into user behaviors in immersive environments.

Figures (7)

• Figure 1: Explainable XR Pipeline Overview: The blue arrows denote the internal calls and flows of Explainable XR, and red arrows denote the inputs of the researcher in our framework. The pipeline initiates by recording the multimodal interactions of the subjects in XR sessions, and importing it into our Action Visual Analyzer via User Prompt Interface. The researcher can perform analytical tasks in our Visual Analytics Interface and optionally utilize our analytics insights during the analysis.
• Figure 2: Action Template Logging Editor and its auto-generated code: Our visual editor streamlines the process of action logging by generating a Unity C# base template code. The user can record subjects' immersive session data with a simple modification of the conditional statement (LogCondition) and the log (Logger.Log) function arguments. The code is generated with a button press in the visual editor, from the user.
• Figure 3: Structure of Logging Function: It conforms to the User Action Descriptor format and internally stores the action data in JSON format, upon invocation.
• Figure 4: Virtuality-agnostic Session Reconstruction: UAD binds each action of a subject with a referent and a scene context. The referent that is a physical entity is inferred through \ref{['para:action_referent_classifier']}, and the virtual entity is logged through gameobject storage. The context point cloud can be generated using the snapshots from a Unity in-application camera, or through a physical XR device camera. The former is mostly used for VR, and the latter, for AR/MR.
• Figure 5: LLM-Analytics Assistance: Given the prompt for the direction of the analysis from the user, we generate analysis direction-tailored Analytics Insights using multimodal LLM agents. The Analysis-of-Interest Markers are associated to every insight, to pin-point the data locations of the referred insights and user's key analytics interests.