Intuitive Human-Drone Collaborative Navigation in Unknown Environments through Mixed Reality

TL;DR

This paper addresses the challenge of intuitive drone navigation in unknown indoor environments by introducing a bidirectional mixed-reality interface that shares spatial representations between an HMD user and an autonomous drone. The approach tightly couples MR visualization with the drone's autonomous planning (via Jump Point Search) and real-time mapping (Octomap with ISDF cues), enabling users to set goals or trajectories while the system autonomously re-plans for safety. User studies in a simulated post-disaster scenario show that MR-assisted navigation reduces cognitive workload (NASA TLX) and increases the explored area compared to traditional FPV control, with substantial gains for both experts and novices. The work demonstrates the practical impact of MR in enhancing situational awareness and safety in mission-critical navigation, suggesting broader applicability to inspection, search-and-rescue, and disaster-response tasks.

Abstract

Considering the widespread integration of aerial robots in inspection, search and rescue, and monitoring tasks, there is a growing demand to design intuitive human-drone interfaces. These aim to streamline and enhance the user interaction and collaboration process during drone navigation, ultimately expediting mission success and accommodating users' inputs. In this paper, we present a novel human-drone mixed reality interface that aims to (a) increase human-drone spatial awareness by sharing relevant spatial information and representations between the human equipped with a Head Mounted Display (HMD) and the robot and (b) enable safer and intuitive human-drone interactive and collaborative navigation in unknown environments beyond the simple command and control or teleoperation paradigm. We validate our framework through extensive user studies and experiments in a simulated post-disaster scenario, comparing its performance against a traditional First-Person View (FPV) control systems. Furthermore, multiple tests on several users underscore the advantages of the proposed solution, which offers intuitive and natural interaction with the system. This demonstrates the solution's ability to assist humans during a drone navigation mission, ensuring its safe and effective execution.

Figures (5)

• Figure 1: Block scheme describing the communication pipeline and the software architecture between the robot and the headset.
• Figure 2: One of the users engaged in the First-Person View modality for robot control.
• Figure 3: NASA TLX Feedback of participants obtained on subjects belonging in Group $1$ (Expert) and Group $2$ (Novice). Labels I, II, III, IV, V, and VI denote Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration.
• Figure 4: Area navigated by drone during MR and FPV modality. The blue area denotes expert Group $1$ users while the red area denotes novice Group $2$ users.
• Figure 5: Exploration paths: (a) Real-scale mesh visualization overlaying the environment as visible through the Hololens, with a representation of the re-planned (red) and the user provided trajectory (green) to the robot, b) Bird's-eye view of the resultant octomap and recorded paths of the collaborative human-drone exploration.