Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Augmented Reality User Interface for Command, Control, and Supervision of Large Multi-Agent Teams

Frank Regal, Chris Suarez, Fabian Parra, Mitch Pryor

TL;DR

This paper presents an AR-HMD based user interface for commanding, controlling, and supervising large multi-agent teams using the AugRE framework. It integrates Robofleet for robot communication, Azure Spatial Anchors for shared localization, and MRTK for AR visualizations to support LOS and NLOS interaction without external hardware. Key contributions include world-positioned AR-labels, live video feedback, distance-aware interaction, GPS-referenced positioning, waypoint navigation, and leader-follower modalities demonstrated in outdoor urban tests. The work demonstrates potential improvements in human-robot collaboration, robustness, and trust for large-scale, heterogeneous teams in challenging environments.

Abstract

Multi-agent human-robot teaming allows for the potential to gather information about various environments more efficiently by exploiting and combining the strengths of humans and robots. In industries like defense, search and rescue, first-response, and others alike, heterogeneous human-robot teams show promise to accelerate data collection and improve team safety by removing humans from unknown and potentially hazardous situations. This work builds upon AugRE, an Augmented Reality (AR) based scalable human-robot teaming framework. It enables users to localize and communicate with 50+ autonomous agents. Through our efforts, users are able to command, control, and supervise agents in large teams, both line-of-sight and non-line-of-sight, without the need to modify the environment prior and without requiring users to use typical hardware (i.e. joysticks, keyboards, laptops, tablets, etc.) in the field. The demonstrated work shows early indications that combining these AR-HMD-based user interaction modalities for command, control, and supervision will help improve human-robot team collaboration, robustness, and trust.

Augmented Reality User Interface for Command, Control, and Supervision of Large Multi-Agent Teams

TL;DR

This paper presents an AR-HMD based user interface for commanding, controlling, and supervising large multi-agent teams using the AugRE framework. It integrates Robofleet for robot communication, Azure Spatial Anchors for shared localization, and MRTK for AR visualizations to support LOS and NLOS interaction without external hardware. Key contributions include world-positioned AR-labels, live video feedback, distance-aware interaction, GPS-referenced positioning, waypoint navigation, and leader-follower modalities demonstrated in outdoor urban tests. The work demonstrates potential improvements in human-robot collaboration, robustness, and trust for large-scale, heterogeneous teams in challenging environments.

Abstract

Multi-agent human-robot teaming allows for the potential to gather information about various environments more efficiently by exploiting and combining the strengths of humans and robots. In industries like defense, search and rescue, first-response, and others alike, heterogeneous human-robot teams show promise to accelerate data collection and improve team safety by removing humans from unknown and potentially hazardous situations. This work builds upon AugRE, an Augmented Reality (AR) based scalable human-robot teaming framework. It enables users to localize and communicate with 50+ autonomous agents. Through our efforts, users are able to command, control, and supervise agents in large teams, both line-of-sight and non-line-of-sight, without the need to modify the environment prior and without requiring users to use typical hardware (i.e. joysticks, keyboards, laptops, tablets, etc.) in the field. The demonstrated work shows early indications that combining these AR-HMD-based user interaction modalities for command, control, and supervision will help improve human-robot team collaboration, robustness, and trust.
Paper Structure (16 sections, 4 figures)

This paper contains 16 sections, 4 figures.

Table of Contents

  1. OVERVIEW
  2. BACKGROUND
  3. USER INTERACTION DETAILS
  4. Situational Awareness
  5. World positioned AR-labels
  6. Close & remote interactions
  7. GPS-referenced world positions
  8. Feedback visualizations
  9. Robot Control
  10. Teleoperation
  11. Robot Commanding
  12. Waypoint Navigation
  13. AR user-to-robot leader-follower control
  14. TEST IMPLEMENTATION
  15. CONCLUSION
  16. ...and 1 more sections

Figures (4)

  • Figure 1: [A] Top-down map view of HoloLens 2 and robotic agent locations. [B] Joystick teleoperation control with FPV robot view for NLOS operation. [C] Autonomous agent navigating to virtual waypoint provided by AR-HMD user. [D] AR-HMD user viewing an agent's navigation path.
  • Figure 2: [A] AR-label used to visualize agent type and location in the environment. Blue NATO symbol represents an unmanned ground vehicle (UGV). [B] AR-label, expanded with click, providing additional information and clickable "Control" and "More" buttons to access more control, command, and supervision functionalities.
  • Figure 3: Holographic waypoint options menu and red holographic marker presented to users to physically define a waypoint navigation goal for autonomous agents.
  • Figure 4: User creating a waypoint navigation path using the waypoint feature menu. Three waypoints were set to command the Clearpath husky pictured in the bottom left to travel to the end of the alleyway.