Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Collaborative Conversation in Safe Multimodal Human-Robot Collaboration

Davide Ferrari, Andrea Pupa, Cristian Secchi

TL;DR

This work addresses the challenge of safe, efficient human-robot collaboration by integrating bidirectional multimodal communication with a safety-aware predictive simulator. The architecture couples voice and gesture channels through a fusion core to produce natural commands, while a Safety Layer enforces ISO/TS 15066 constraints via online velocity scaling and a predictive simulator that preemptively guides operator dialogue. Experimental validation on a UR10e against a state-of-the-art baseline shows significant gains: ~23% faster task completion and ~50% less robot downtime, along with higher user-satisfaction scores. The approach advances HRC by treating safety as an integral part of conversation, enabling proactive, human-centered control in shared workspaces.

Abstract

In the context of Human-Robot Collaboration (HRC), it is crucial that the two actors are able to communicate with each other in a natural and efficient manner. The absence of a communication interface is often a cause of undesired slowdowns. On one hand, this is because unforeseen events may occur, leading to errors. On the other hand, due to the close contact between humans and robots, the speed must be reduced significantly to comply with safety standard ISO/TS 15066. In this paper, we propose a novel architecture that enables operators and robots to communicate efficiently, emulating human-to-human dialogue, while addressing safety concerns. This approach aims to establish a communication framework that not only facilitates collaboration but also reduces undesired speed reduction. Through the use of a predictive simulator, we can anticipate safety-related limitations, ensuring smoother workflows, minimizing risks, and optimizing efficiency. The overall architecture has been validated with a UR10e and compared with a state of the art technique. The results show a significant improvement in user experience, with a corresponding 23% reduction in execution times and a 50% decrease in robot downtime.

Collaborative Conversation in Safe Multimodal Human-Robot Collaboration

TL;DR

This work addresses the challenge of safe, efficient human-robot collaboration by integrating bidirectional multimodal communication with a safety-aware predictive simulator. The architecture couples voice and gesture channels through a fusion core to produce natural commands, while a Safety Layer enforces ISO/TS 15066 constraints via online velocity scaling and a predictive simulator that preemptively guides operator dialogue. Experimental validation on a UR10e against a state-of-the-art baseline shows significant gains: ~23% faster task completion and ~50% less robot downtime, along with higher user-satisfaction scores. The approach advances HRC by treating safety as an integral part of conversation, enabling proactive, human-centered control in shared workspaces.

Abstract

In the context of Human-Robot Collaboration (HRC), it is crucial that the two actors are able to communicate with each other in a natural and efficient manner. The absence of a communication interface is often a cause of undesired slowdowns. On one hand, this is because unforeseen events may occur, leading to errors. On the other hand, due to the close contact between humans and robots, the speed must be reduced significantly to comply with safety standard ISO/TS 15066. In this paper, we propose a novel architecture that enables operators and robots to communicate efficiently, emulating human-to-human dialogue, while addressing safety concerns. This approach aims to establish a communication framework that not only facilitates collaboration but also reduces undesired speed reduction. Through the use of a predictive simulator, we can anticipate safety-related limitations, ensuring smoother workflows, minimizing risks, and optimizing efficiency. The overall architecture has been validated with a UR10e and compared with a state of the art technique. The results show a significant improvement in user experience, with a corresponding 23% reduction in execution times and a 50% decrease in robot downtime.
Paper Structure (12 sections, 6 equations, 4 figures, 3 tables, 2 algorithms)

This paper contains 12 sections, 6 equations, 4 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Proposed Architecture
  4. Multimodal Fusion
  5. Safety
  6. Safety Layer
  7. Online Safety
  8. Predictive Simulator
  9. Experimental Validation
  10. Implementation Details
  11. Analysis of the Results
  12. Conclusions and Future Work

Figures (4)

  • Figure 1: Proposed Architecture
  • Figure 2: Setup of the Experiment
  • Figure 3: Questionnaire Results
  • Figure 4: Execution Times and Robot Downtime