ARTHUR: Authoring Human-Robot Collaboration Processes with Augmented Reality using Hybrid User Interfaces

TL;DR

ARTHUR addresses the challenge of authoring AR-guided human-robot collaboration (HRC) by introducing an open-source tool that enables in-situ AR authoring through a hybrid user interface. It provides three design components—Feedback, Actions, and Conditions—along with trackers/anchors, an MQTT-based service architecture, and a UR5e robot adapter to integrate robot data and control. The authors demonstrate ARTHUR's utility by replicating representative HRC scenarios and conducting a qualitative expert evaluation to assess usability and the viability of distributing authoring across desktop, tablet, and HMD devices. This work advances practical AR-HRC development by enabling rapid design, testing, and deployment of AR-guided workflows across multiple devices, with clear paths for extensibility and integration with PLM data and robot controllers.

Abstract

While augmented reality shows promise for supporting human-robot collaboration, creating such interactive systems still poses great challenges. Addressing this, we introduce ARTHUR, an open-source authoring tool for augmented reality-supported human-robot collaboration. ARTHUR supports 20 types of multi-modal feedback to convey robot, task, and system state, 10 actions that enable the user to control the robot and system, and 18 conditions for feedback customization and triggering of actions. By combining these elements, users can create interaction spaces, controls, and information visualizations in augmented reality for collaboration with robot arms. With ARTHUR, we propose to combine desktop interfaces and touchscreen devices for effective authoring, with head-mounted displays for testing and in-situ refinements. To demonstrate the general applicability of ARTHUR for human-robot collaboration scenarios, we replicate representative examples from prior work. Further, in an evaluation with five participants, we reflect on the usefulness of our hybrid user interface approach and the provided functionality, highlighting directions for future work.

Figures (7)

• Figure 1: ARTHUR is an authoring tool for augmented reality-supported human-robot collaboration. It supports (1) creating the initial system configuration in the web interface (PC) (left), (2) refining the setup in-situ using a hybrid of web interface (tablet) and AR interface (HMD) (center), and (3) testing and using the authored system on the AR interface (HMD) (right).
• Figure 2: Figure showing all interfaces and services and which device they live on. The boundary indicated by a dotted line highlights what is included in ARTHUR and what must be provided, e.g., the robot program is not created in ARTHUR or is a direct part of it but communicates with the system through the robot adapter service.
• Figure 3: Left and middle show screenshots from the web interface, where the rendering is the same across devices (PC and tablet). Right shows the AR interface after clicking "Set/highlight anchors" (G). The pages are similar for all design components (\ref{['fig:building_blocks']}), with a page for viewing, creating, and editing feedback, actions, and conditions. A position can also be viewed in AR (H) and set (J). The sub-menus for authoring is found under (A), it is possible to create a new system setup from the "Workstations tap" (E), edit agents, trackers, tools, parts, and tasks (B), get a status off all services (F), generate "fake data", e.g. path, waypoints, zones, and messages, which can be used to test the appearance of some visualizations in current lack of any real data (C), and finally customize the interface, e.g. by showing sliders for number input instead of raw number input (D).
• Figure 4: Overview of all feedback, actions, and conditions currently implemented in ARTHUR.
• Figure 5: Scenario 1 hietanen2020ar replicated in ARTHUR. The left part shows the view from the AR interface, with each design component annotated with a number. The right part depicts a list of feedback from the web interface, with corresponding numbers annotated, a list of created conditions, and finally a list of actions.