Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Alternative Interfaces for Human-initiated Natural Language Communication and Robot-initiated Haptic Feedback: Towards Better Situational Awareness in Human-Robot Collaboration

Callum Bennie, Bridget Casey, Cecile Paris, Dana Kulic, Brendan Tidd, Nicholas Lawrance, Alex Pitt, Fletcher Talbot, Jason Williams, David Howard, Pavan Sikka, Hashini Senaratne

TL;DR

The paper addresses the gap in situational awareness for human-robot collaboration by introducing two alternative interfaces: a natural-language speech interface and a forearm-based haptic feedback wearable for Spot robots. It details a pipeline that combines Whisper for speech-to-text, a GPT-3-based interpreter with retrieval-augmented prompting, and Silero for text-to-speech, producing JSON responses to control robots and query status, with a design emphasis on offline viability and multi-robot scalability. It also presents a ten-motor, forearm vibrotactile array controlled via ROS to convey robot status through customizable patterns, including a comprehensive GUI for configuring event-driven patterns and priorities. Preliminary user evaluations indicate that both interfaces support intuitive interaction and improved awareness, while highlighting latency, pattern discrimination, and portability as areas for improvement. Collectively, the work demonstrates the potential of multimodal, human-centered interfaces to enhance responsiveness and reduce cognitive load in real-time human-robot teams, with implications for disaster response and collaborative automation.

Abstract

This article presents an implementation of a natural-language speech interface and a haptic feedback interface that enables a human supervisor to provide guidance to, request information, and receive status updates from a Spot robot. We provide insights gained during preliminary user testing of the interface in a realistic robot exploration scenario.

Alternative Interfaces for Human-initiated Natural Language Communication and Robot-initiated Haptic Feedback: Towards Better Situational Awareness in Human-Robot Collaboration

TL;DR

The paper addresses the gap in situational awareness for human-robot collaboration by introducing two alternative interfaces: a natural-language speech interface and a forearm-based haptic feedback wearable for Spot robots. It details a pipeline that combines Whisper for speech-to-text, a GPT-3-based interpreter with retrieval-augmented prompting, and Silero for text-to-speech, producing JSON responses to control robots and query status, with a design emphasis on offline viability and multi-robot scalability. It also presents a ten-motor, forearm vibrotactile array controlled via ROS to convey robot status through customizable patterns, including a comprehensive GUI for configuring event-driven patterns and priorities. Preliminary user evaluations indicate that both interfaces support intuitive interaction and improved awareness, while highlighting latency, pattern discrimination, and portability as areas for improvement. Collectively, the work demonstrates the potential of multimodal, human-centered interfaces to enhance responsiveness and reduce cognitive load in real-time human-robot teams, with implications for disaster response and collaborative automation.

Abstract

This article presents an implementation of a natural-language speech interface and a haptic feedback interface that enables a human supervisor to provide guidance to, request information, and receive status updates from a Spot robot. We provide insights gained during preliminary user testing of the interface in a realistic robot exploration scenario.
Paper Structure (14 sections, 3 figures, 1 table)

This paper contains 14 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Human-initiated Natural Language Communication with Robots
  3. Implementation of Speech Interface
  4. System Overview
  5. Prompt creation
  6. Preliminary User Evaluation
  7. Robot-initiated Haptic Feedback
  8. Implementation of Haptic Interface
  9. System Overview
  10. Wearable Design
  11. Configuration Graphical User Interface
  12. Preliminary User Evaluation
  13. Future Directions
  14. Conclusion

Figures (3)

  • Figure 1: System overview with sample user requests and GPT responses
  • Figure 2: The haptic feedback wearable (a) worn at forearm of the user while using the graphical user interface to supervise robots, and (b) a close look of motor configuration, 3D printed holder and velcro straps used to secure the wearable to the user's forearm.
  • Figure 3: The graphical interface for configuring haptic patterns for different status updates from multiple robots