Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Does ChatGPT and Whisper Make Humanoid Robots More Relatable?

Xiaohui Chen, Katherine Luo, Trevor Gee, Mahla Nejati

TL;DR

This work investigates whether integrating Whisper ASR and ChatGPT with the Pepper humanoid robot enhances relatability and interaction quality. The Pepper-GPT architecture combines a Whisper-based speech recognizer with GPT-3.5-turbo to produce either physical actions or natural language responses, coordinated by a PepperController. Evaluation shows Whisper delivers the best speech recognition performance (WER around 1.716% and processing time ~2.639 s) across diverse accents, and a 15-participant user study reports generally positive experiences, with 60% rating interactions as excellent. The study identifies remaining challenges such as multilingual support and facial tracking, offering a roadmap for more natural, engaging hri systems with greater practical impact.

Abstract

Humanoid robots are designed to be relatable to humans for applications such as customer support and helpdesk services. However, many such systems, including Softbank's Pepper, fall short because they fail to communicate effectively with humans. The advent of Large Language Models (LLMs) shows the potential to solve the communication barrier for humanoid robotics. This paper outlines the comparison of different Automatic Speech Recognition (ASR) APIs, the integration of Whisper ASR and ChatGPT with the Pepper robot and the evaluation of the system (Pepper-GPT) tested by 15 human users. The comparison result shows that, compared to the Google ASR and Google Cloud ASR, the Whisper ASR performed best as its average Word Error Rate (1.716%) and processing time (2.639 s) are both the lowest. The participants' usability investigations show that 60% of the participants thought the performance of the Pepper-GPT was "excellent", while the rest rated this system as "good" in the subsequent experiments. It is proved that while some problems still need to be overcome, such as the robot's multilingual ability and facial tracking capacity, users generally responded positively to the system, feeling like talking to an actual human.

Does ChatGPT and Whisper Make Humanoid Robots More Relatable?

TL;DR

This work investigates whether integrating Whisper ASR and ChatGPT with the Pepper humanoid robot enhances relatability and interaction quality. The Pepper-GPT architecture combines a Whisper-based speech recognizer with GPT-3.5-turbo to produce either physical actions or natural language responses, coordinated by a PepperController. Evaluation shows Whisper delivers the best speech recognition performance (WER around 1.716% and processing time ~2.639 s) across diverse accents, and a 15-participant user study reports generally positive experiences, with 60% rating interactions as excellent. The study identifies remaining challenges such as multilingual support and facial tracking, offering a roadmap for more natural, engaging hri systems with greater practical impact.

Abstract

Humanoid robots are designed to be relatable to humans for applications such as customer support and helpdesk services. However, many such systems, including Softbank's Pepper, fall short because they fail to communicate effectively with humans. The advent of Large Language Models (LLMs) shows the potential to solve the communication barrier for humanoid robotics. This paper outlines the comparison of different Automatic Speech Recognition (ASR) APIs, the integration of Whisper ASR and ChatGPT with the Pepper robot and the evaluation of the system (Pepper-GPT) tested by 15 human users. The comparison result shows that, compared to the Google ASR and Google Cloud ASR, the Whisper ASR performed best as its average Word Error Rate (1.716%) and processing time (2.639 s) are both the lowest. The participants' usability investigations show that 60% of the participants thought the performance of the Pepper-GPT was "excellent", while the rest rated this system as "good" in the subsequent experiments. It is proved that while some problems still need to be overcome, such as the robot's multilingual ability and facial tracking capacity, users generally responded positively to the system, feeling like talking to an actual human.
Paper Structure (19 sections, 1 equation, 13 figures, 1 table)

This paper contains 19 sections, 1 equation, 13 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Speech Recognition
  4. Natural Language Processing
  5. Methodology
  6. BlackBox
  7. Speech Recognition Module
  8. GPT Module
  9. PepperController
  10. Data Transmission
  11. Evaluation
  12. Speech Recognition Evaluation
  13. Measurement Metric
  14. Dataset
  15. Results
  16. ...and 4 more sections

Figures (13)

  • Figure 1: The Workflow of Pepper-GPT.
  • Figure 2: Pseudo code for GPT module.
  • Figure 3: Performance of Average WER across different accents.
  • Figure 4: Performance of Average Recognition Time across different accents.
  • Figure 5: Performance of Average WER on different daily dialogues.
  • ...and 8 more figures