Spot-On: A Mixed Reality Interface for Multi-Robot Cooperation

TL;DR

Spot-On introduces a mixed-reality interface for coordinating multiple Spot robots to perform object manipulation and infrastructure tasks in real-world layouts. It uses a scene graph as the central representation and a Unity-based MR/VR app to enable intuitive selection and command delegation, with per-robot servers for state synchronization. The approach supports drawers, swing doors, light switches, and object grasping through integrated perception (OpenMask3D, YOLOv8) and manipulation (AnyGrasp) capabilities, enabling collaborative inspection and manipulation. The study demonstrates usability and a positive reception, highlighting potential for scalable MR-based teleoperation of multi-robot teams in cluttered environments.

Abstract

Recent progress in mixed reality (MR) and robotics is enabling increasingly sophisticated forms of human-robot collaboration. Building on these developments, we introduce a novel MR framework that allows multiple quadruped robots to operate in semantically diverse environments via a MR interface. Our system supports collaborative tasks involving drawers, swing doors, and higher-level infrastructure such as light switches. A comprehensive user study verifies both the design and usability of our app, with participants giving a "good" or "very good" rating in almost all cases. Overall, our approach provides an effective and intuitive framework for MR-based multi-robot collaboration in complex, real-world scenarios.

Figures (10)

• Figure 1: Spot-On. We introduce Spot-On, a Mixed-Reality application that allows Multi-Robot Control and Collaboration
• Figure 2: Spot-On Overview. Our user friendly top-level mixed-reality interface allows for intuitive robotic control. Starting from the main menu, the user opens the scene view (I.). The user is presented with an interactable 3D reconstruction of the scene. By hovering objects in the scene and clicking them, objects can be selected (II.). Now, object specific actions are highlighted and can be selected at will (III.). Once the action is selected in the virtual user interface, the robots automatically start performing the task in the real scene (IV.).
• Figure 3: Networking. The networking infrastructure relies on one server per robot. The HoloLens posts commands for the robots to execute, while the robots post their states and changes they have made in the environment. We query these changes, so that the scene graph representation in the user interface stays up-to-date.
• Figure 4: Scene and Menu. Figure shows scene (left) and start menu (right) as they are seen by user in the MR/VR device.
• Figure 5: Interface. Figure depicts the app's response to user interaction by hovering and clicking on different object types. The first row shows hovering events (popups), and the second row shows clicking events (prompts). From left to right, interactions are with a lamp, drawer, and light switch.