Investigating Mixed Reality for Communication Between Humans and Mobile Manipulators

TL;DR

This study extends mixed reality-based intent communication to mobile manipulators (MR-HRC v2), using holograms to convey upcoming robot actions and states to human teammates. Implemented on ROS TIAGo++ with YOLOv5-based perception and Hololens 2 visualization, the architecture demonstrates that anticipatory MR cues can reduce human interruptions and improve task fluency without increasing total collaboration time. A 20-participant user study in a warehouse-like scenario shows the human restocking task is completed faster when MR guidance is present, driven by increased proactive human engagement and fewer failed interactions. The work suggests MR-based communicative overlays can enhance safety, predictability, and efficiency in human-robot teams, particularly in mobile, dynamic collaborative settings.

Abstract

This article investigates mixed reality (MR) to enhance human-robot collaboration (HRC). The proposed solution adopts MR as a communication layer to convey a mobile manipulator's intentions and upcoming actions to the humans with whom it interacts, thus improving their collaboration. A user study involving 20 participants demonstrated the effectiveness of this MR-focused approach in facilitating collaborative tasks, with a positive effect on overall collaboration performances and human satisfaction.

Figures (8)

• Figure 1: Fig. \ref{['fig:hrc']} illustrates the scenario of human-robot collaboration, setting the stage for understanding the context. Following this, Fig. \ref{['fig:dt']} offers a view from the human perspective within mixed reality. This perspective demonstrates how the handover process will occur by conveying the robot's intention, thereby ensuring human safety.
• Figure 2: A detailed overview of MR-HRC v2, with the two main blocks highlighted by the corresponding colours.
• Figure 3: Fig. \ref{['holo1']} and \ref{['holo2']} show the holographic communication from the human perspective, respectively, for robot navigation and manipulation. Fig. \ref{['holo3']} and \ref{['ext1']} present the results of the perception module, with the bottles detected by YOLOv5m and the corresponding point cloud acquired by the ZED2 camera mounted on the robot.
• Figure 4: Bird's-eye view of the collaborative workspace, displaying labelled points of interest alongside the workspace dimensions.
• Figure 5: Time metrics (in seconds) observed during trials, in the two experimental conditions. Fig. \ref{['fig::overall-task-time']} depicts the total time needed to complete the collaboration, measured once the human and the robot had both completed their tasks. Conversely, Fig. \ref{['fig::human-task-time']} depicts the time taken by participants to complete their restocking task, measured once the human had put all twelve bottles on shelf 3 in the correct order. The small circles depicted in Fig. \ref{['fig::human-task-time']} highlight an outlier in the distribution.