Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Real-Time Imitation of Human Head Motions, Blinks and Emotions by Nao Robot: A Closed-Loop Approach

Keyhan Rayati, Amirhossein Feizi, Alireza Beigy, Pourya Shahverdi, Mehdi Tale Masouleh, Ahmad Kalhor

TL;DR

The paper addresses real-time imitation of human head motion, blinks, and emotions by a Nao robot to improve human–robot interaction. It combines MediaPipe-based pose/face mesh analysis with DeepFace-based emotion recognition and a closed-loop control using the Naoqi API to mirror human cues and adapt responses. The approach yields high accuracy in head imitation (e.g., $R^2$ values approaching $0.99$ for yaw and pitch) and reliable blink and emotion-driven interactions, aided by SVR-based pitch constraints to maintain safe robot motion. The work demonstrates potential benefits for children with autism by enabling more natural, emotionally aware interactions.

Abstract

This paper introduces a novel approach for enabling real-time imitation of human head motion by a Nao robot, with a primary focus on elevating human-robot interactions. By using the robust capabilities of the MediaPipe as a computer vision library and the DeepFace as an emotion recognition library, this research endeavors to capture the subtleties of human head motion, including blink actions and emotional expressions, and seamlessly incorporate these indicators into the robot's responses. The result is a comprehensive framework which facilitates precise head imitation within human-robot interactions, utilizing a closed-loop approach that involves gathering real-time feedback from the robot's imitation performance. This feedback loop ensures a high degree of accuracy in modeling head motion, as evidenced by an impressive R2 score of 96.3 for pitch and 98.9 for yaw. Notably, the proposed approach holds promise in improving communication for children with autism, offering them a valuable tool for more effective interaction. In essence, proposed work explores the integration of real-time head imitation and real-time emotion recognition to enhance human-robot interactions, with potential benefits for individuals with unique communication needs.

Real-Time Imitation of Human Head Motions, Blinks and Emotions by Nao Robot: A Closed-Loop Approach

TL;DR

The paper addresses real-time imitation of human head motion, blinks, and emotions by a Nao robot to improve human–robot interaction. It combines MediaPipe-based pose/face mesh analysis with DeepFace-based emotion recognition and a closed-loop control using the Naoqi API to mirror human cues and adapt responses. The approach yields high accuracy in head imitation (e.g., values approaching for yaw and pitch) and reliable blink and emotion-driven interactions, aided by SVR-based pitch constraints to maintain safe robot motion. The work demonstrates potential benefits for children with autism by enabling more natural, emotionally aware interactions.

Abstract

This paper introduces a novel approach for enabling real-time imitation of human head motion by a Nao robot, with a primary focus on elevating human-robot interactions. By using the robust capabilities of the MediaPipe as a computer vision library and the DeepFace as an emotion recognition library, this research endeavors to capture the subtleties of human head motion, including blink actions and emotional expressions, and seamlessly incorporate these indicators into the robot's responses. The result is a comprehensive framework which facilitates precise head imitation within human-robot interactions, utilizing a closed-loop approach that involves gathering real-time feedback from the robot's imitation performance. This feedback loop ensures a high degree of accuracy in modeling head motion, as evidenced by an impressive R2 score of 96.3 for pitch and 98.9 for yaw. Notably, the proposed approach holds promise in improving communication for children with autism, offering them a valuable tool for more effective interaction. In essence, proposed work explores the integration of real-time head imitation and real-time emotion recognition to enhance human-robot interactions, with potential benefits for individuals with unique communication needs.
Paper Structure (12 sections, 18 equations, 9 figures)

This paper contains 12 sections, 18 equations, 9 figures.

Table of Contents

  1. Introduction
  2. Human Imitation and Interaction Framework
  3. Environment Setup
  4. Head Imitation
  5. Head Yaw Estimation
  6. Head Pitch Estimation
  7. Blink Detection
  8. Defining Eye Regions
  9. Blink Thresholding
  10. Emotion Detection
  11. Results and Experiments
  12. Conclusions

Figures (9)

  • Figure 1: Framework overview: Real-time head motion imitation and emotion detection.
  • Figure 2: Interaction diagram of MediaPipe, Naoqi API, and framework.
  • Figure 3: Real-Time Imitation System's Closed-Loop. (a) Capturing User's Face with Webcam, (b) Landmark Calculation Using MediaPipe, (c) Facial Landmarks Overlay, (d) Angle Calculations for Head Pitch and Yaw, (e) Predictive Angle Calculation for Unobserved Data, (f) Nao Robot Integration for Angle Execution, with Error Calculation Through Feedback.
  • Figure 4: Eye closure interactions, (a) Both eyes open, (b) Right eye closed, (c) Left eye closed and (d) Both eyes closed.
  • Figure 5: Emotion-to-Response Mapping for the Nao robot using DeepFace analysis.
  • ...and 4 more figures