Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Emotional Speech Synthesis for Companion Robot to Imitate Professional Caregiver Speech

Takeshi Homma, Qinghua Sun, Takuya Fujioka, Ryuta Takawaki, Eriko Ankyu, Kenji Nagamatsu, Daichi Sugawara, Etsuko T. Harada

TL;DR

A speech synthesis method for imitating the emotional states in human speech that requires less manual effort to adjust the emotion of the synthesized speech and shows that listening to the samples made the participants feel more active in the early morning and calmer in the middle of the night.

Abstract

When people try to influence others to do something, they subconsciously adjust their speech to include appropriate emotional information. In order for a robot to influence people in the same way, the robot should be able to imitate the range of human emotions when speaking. To achieve this, we propose a speech synthesis method for imitating the emotional states in human speech. In contrast to previous methods, the advantage of our method is that it requires less manual effort to adjust the emotion of the synthesized speech. Our synthesizer receives an emotion vector to characterize the emotion of synthesized speech. The vector is automatically obtained from human utterances by using a speech emotion recognizer. We evaluated our method in a scenario when a robot tries to regulate an elderly person's circadian rhythm by speaking to the person using appropriate emotional states. For the target speech to imitate, we collected utterances from professional caregivers when they speak to elderly people at different times of the day. Then we conducted a subjective evaluation where the elderly participants listened to the speech samples generated by our method. The results showed that listening to the samples made the participants feel more active in the early morning and calmer in the middle of the night. This suggests that the robot may be able to adjust the participants' circadian rhythm and that the robot can potentially exert influence similarly to a person.

Emotional Speech Synthesis for Companion Robot to Imitate Professional Caregiver Speech

TL;DR

A speech synthesis method for imitating the emotional states in human speech that requires less manual effort to adjust the emotion of the synthesized speech and shows that listening to the samples made the participants feel more active in the early morning and calmer in the middle of the night.

Abstract

When people try to influence others to do something, they subconsciously adjust their speech to include appropriate emotional information. In order for a robot to influence people in the same way, the robot should be able to imitate the range of human emotions when speaking. To achieve this, we propose a speech synthesis method for imitating the emotional states in human speech. In contrast to previous methods, the advantage of our method is that it requires less manual effort to adjust the emotion of the synthesized speech. Our synthesizer receives an emotion vector to characterize the emotion of synthesized speech. The vector is automatically obtained from human utterances by using a speech emotion recognizer. We evaluated our method in a scenario when a robot tries to regulate an elderly person's circadian rhythm by speaking to the person using appropriate emotional states. For the target speech to imitate, we collected utterances from professional caregivers when they speak to elderly people at different times of the day. Then we conducted a subjective evaluation where the elderly participants listened to the speech samples generated by our method. The results showed that listening to the samples made the participants feel more active in the early morning and calmer in the middle of the night. This suggests that the robot may be able to adjust the participants' circadian rhythm and that the robot can potentially exert influence similarly to a person.

Paper Structure

This paper contains 22 sections, 5 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Pipeline
  4. Training of Emotional Speech Synthesis Model
  5. Speech Synthesis Conditioned by Emotion of Target Speech
  6. Emotional Statistics Calculation of Target Speech
  7. Emotional Input for Speech Synthesis
  8. Pipeline Implementation
  9. Speech Emotion Recognizer
  10. Speech Synthesizer
  11. Voice Actor Recording
  12. Synthesizer Model Training
  13. Collection of Caregiver Speech
  14. Speech Collection Method
  15. Results of Emotion Recognition
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Pipeline of emotional speech synthesizer.
  • Figure 2: Emotion recognition results of caregivers' voices (OW: Opening Words, PU: Positive Utterance, NU: Negative Utterance).
  • Figure 3: Input emotional parameters for speech synthesis.
  • Figure 4: Experimental robot.
  • Figure 5: Subjective evaluation results of emotional voices and baselines. Error bars indicate 95% confidence intervals. Asterisks indicate significant differences from 50, i.e., the tie-judgment level (One sample $t$-test with two-tailed hypothesis; *$p<0.05$, **$p<0.01$).