MRNaB: Mixed Reality-based Robot Navigation Interface using Optical-see-through MR-beacons

TL;DR

MRNaB tackles the limitations of 2D robot navigation by introducing a mixed reality interface that uses persistent MR-beacons created with air-tap gestures. The system integrates Hololens 2 with ROS 2, featuring four functions (Add, Move, Select, Delete) and a database to support multi-destination visualization and persistent goals. Experimental results show MRNaB reduces the number of required beacon actions and navigations compared to a 2D baseline, with longer per-action placement times but improved user confidence and preference for MR visualization. The approach enables more intuitive, real-world-oriented robot navigation with potential benefits for household and delivery tasks, though occlusion and far-distance setup remain areas for future improvement.

Abstract

Recent advancements in robotics have led to the development of numerous interfaces to enhance the intuitiveness of robot navigation. However, the reliance on traditional 2D displays imposes limitations on the simultaneous visualization of information. Mixed Reality (MR) technology addresses this issue by enhancing the dimensionality of information visualization, allowing users to perceive multiple pieces of information concurrently. This paper proposes the Mixed Reality-based Robot Navigation Interface using an Optical-see-through MR-beacons (MRNaB), a novel approach that uses MR-beacons created with an ``air tap'', situated in the real world. This beacon is persistent, enabling multi-destination visualization and functioning as a signal transmitter for robot navigation, eliminating the need for repeated navigation inputs. Our system is mainly constructed into four primary functions: ``Add'', ``Move'', ``Delete'', and ``Select''. These allow for the addition of MR-beacons, location movement, its deletion, and the selection of MR-beacons for navigation purposes, respectively. To validate the effectiveness, we conducted comprehensive experiments comparing MRNaB with traditional 2D navigation systems. The results show significant improvements in user performance, both objectively and subjectively, confirming that the MRNaB enhances navigation efficiency and user experience. For additional material, please check: https://mertcookimg.github.io/mrnab

Figures (17)

• Figure 1: MRNaB Concept. (1-a) shows the initial state where there is no MR-beacon on the floor. Next, (1-b) MR-beacon will be set to a certain location on the floor where even after leaving the project or restarting it, the MR-beacon will still be there and be used for robot navigation. (2-a) For navigation, the beacon just needs to be clicked by the user, then (2-b) the navigation will start and (2-c) the robot will move to the desired place.
• Figure 2: System Design Diagram. The user interacts with Hololens 2 by using gestures to access the hand menu and interact with the MR beacon to access the beacon functions. Our system also has a database that saves the information (Pose and ID) of the robot and uses it during the first co-localization. Hololens 2 receives robot data while sending the log and goal pose to the ROS 2 side. ROS 2 has a log recorder and TF converter to convert robot and map information to only robot information. Goal pose will be sent directly using the Kachaka gprc ROS 2 bridge to the "Kachaka" robot.
• Figure 3: Hand Menu. (a) shows main menu which is shown when user does ”Hand Constraint Palm Up” movement. (b) shows the beacon menu after pressing the beacon button in main menu.
• Figure 4: Add Mode Process. (a) shows the initial state where there is no MR-beacon on the floor. (b) When user does the "air tap" movement to the floor, MR-beacon will be generated and location setting starts. (c) As user drags the pointer, MR-beacon will follow the pointer. Once "air tap" is released, MR-beacon will fix the location and start the direction setting. (d) To fix the direction, user has to do one more "air tap".
• Figure 5: Move Mode Process. (a) shows the initial state where there is an MR-beacon on the floor. (b) When user does the "air tap" movement to the MR-beacon, MR-beacon will be generated and the location setting starts. (c) As user drags the pointer, MR-beacon will follow the pointer. Once "air tap" is released, MR-beacon will fix the location and start the direction setting. (d) To fix the direction, user has to do one more "air tap".