MemoryPods: Enhancing Asynchronous Communication in Extended Reality

TL;DR

The paper addresses the challenge of effective asynchronous communication in XR by introducing MemoryPods, which record, annotate, and replay interactions with spatial-temporal fidelity and AI-driven narrative summaries. The system uses contextual annotations, a spatial anchor, body-movement tracking, environment recording, and narrative abstraction to provide multi-perspective replay and reduce cognitive load. An empirical study in remote maintenance demonstrates improved comprehension and task recall compared with traditional text and video, with XR real-scale offering the best usability and spatial accuracy. The approach holds promise for domains such as education, healthcare, remote collaboration, and training, though limitations in 3D reconstruction fidelity are acknowledged and avenues for future enhancement are suggested.

Abstract

Asynchronous communication has become increasingly essential in the context of extended reality (XR), enabling users to interact and share information immersively without the constraints of simultaneous engagement. However, current XR systems often struggle to support effective asynchronous interactions, mainly due to limitations in contextual replay and navigation. This paper aims to address these limitations by introducing a novel system that enhances asynchronous communication in XR through the concept of MemoryPods, which allow users to record, annotate, and replay interactions with spatial and temporal accuracy. MemoryPods also feature AI-driven summarisation to ease cognitive load. A user evaluation conducted in a remote maintenance scenario demonstrated significant improvements in comprehension, highlighting the system's potential to transform collaboration in XR. The findings suggest broad applicability of the proposed system across various domains, including direct messaging, healthcare, education, remote collaboration, and training, offering a promising solution to the complexities of asynchronous communication in immersive environments.

Figures (10)

• Figure 1: Workflow of the proposed system for enhancing asynchronous communication in XR environments. The process begins with real-world data recording through an XR device, capturing video, tracked items, and annotations. The data is sent to a cloud server for processing, where LLMs aid in data extraction and augmentation. Playback is offered in two forms: a miniature variant or a real-scale variant, both accessible in augmented reality.
• Figure 2: Annotation procedure. The user places a spatio-temporal annotation to indicate when and where this component should be acquired.
• Figure 3: Original environment and the 3D Reconstruction of its recording. The large blue sphere represents the head of the user, the smaller spheres indicate their hands, while the faded triangle showcases their field of view.
• Figure 4: Narrative abstraction of the selected recording using LLMs.
• Figure 5: Illustration of two XR recordings in their miniature scale version, placed simultaneously within the environment. This feature enables users to review and manage multiple recorded processes concurrently, facilitating multitasking within the immersive space.