Visualizing Causality in Mixed Reality for Manual Task Learning: An Exploratory Study

TL;DR

This study investigates how visualizing causality in Mixed Reality (MR) affects manual task learning. It introduces a three-level hierarchy of causality—event, interaction, and gesture—and compares four visualization conditions through a two-phase study with 48 participants performing a camera-stand assembly task. Results indicate that incorporating full causal visualization improves testing performance and task completion speed, with gesture-level detail offering the strongest memory and execution gains, albeit at the cost of longer learning. The findings offer design guidance for MR-based instruction and point to dynamic, intention-aligned causality visualization as a promising avenue for future MR training systems.

Abstract

Mixed Reality (MR) is gaining prominence in manual task skill learning due to its in-situ, embodied, and immersive experience. To teach manual tasks, current methodologies break the task into hierarchies (tasks into subtasks) and visualize the current subtask and future in terms of causality. Existing psychology literature also shows that humans learn tasks by breaking them into hierarchies. In order to understand the design space of information visualized to the learner for better task understanding, we conducted a user study with 48 users. The study was conducted using a complex assembly task, which involves learning of both actions and tool usage. We aim to explore the effect of visualization of causality in the hierarchy for manual task learning in MR by four options: no causality, event level causality, interaction level causality, and gesture level causality. The results show that the user understands and performs best when all the level of causality is shown to the user. Based on the results, we further provide design recommendations and in-depth discussions for future manual task learning systems.

Figures (14)

• Figure 1: A three-level hierarchy for causality in a manual task. The three levels are event, interaction, and gesture, respectively. Horizontally, each node is connected by causal relations, explaining the cause and effect among them. E.g., the occurrence of event 1 leads to the occurrence of event 2. A node from a higher level consists of one or multiple children at the lower level. We aim to differentiate different levels of causality and study their effects in manual task learning.
• Figure 2: Our methodology to visualize the causality in the manual task at three levels. In the Cause column, we will show the demonstration of (event, interaction, or gesture of) the current step, which is the cause. In the Effect column, we will show the corresponding level of demonstration of the future step that is causally related to the current step.
• Figure 3: The camera setup to be assembled and the tools to be used in our test bed.
• Figure 4: The printer setup in our test bed.
• Figure 5: UIs that were shown to 4 different groups and AR animation. a) shows the event, interaction, and affordance/gesture, b) shows the event and interaction level relation, c) shows the event level causal relation and finally d) shows the graph and no causal relation.