Integrating Persian Lip Reading in Surena-V Humanoid Robot for Human-Robot Interaction
Ali Farshian Abbasi, Aghil Yousefi-Koma, Soheil Dehghani Firouzabadi, Parisa Rashidi, Alireza Naeini
TL;DR
This work tackles lip-reading for Persian in humanoid-robot interaction to improve communication when verbal cues are limited. It evaluates two pipelines: an indirect approach using facial landmark tracking and a direct approach using CNNs and LSTMs to process raw video data, with real-time deployment on the Surena-V robot. The direct LSTM method achieves 89% accuracy on test data and is integrated into Surena-V for responsive behavior, highlighting the practical viability of video-based speech recognition in robotic HRI. The study also notes dataset collection challenges and suggests future work to enhance motion modeling and expand vocabulary and speaker diversity to improve generalization in real-world environments.
Abstract
Lip reading is vital for robots in social settings, improving their ability to understand human communication. This skill allows them to communicate more easily in crowded environments, especially in caregiving and customer service roles. Generating a Persian Lip-reading dataset, this study integrates Persian lip-reading technology into the Surena-V humanoid robot to improve its speech recognition capabilities. Two complementary methods are explored, an indirect method using facial landmark tracking and a direct method leveraging convolutional neural networks (CNNs) and long short-term memory (LSTM) networks. The indirect method focuses on tracking key facial landmarks, especially around the lips, to infer movements, while the direct method processes raw video data for action and speech recognition. The best-performing model, LSTM, achieved 89\% accuracy and has been successfully implemented into the Surena-V robot for real-time human-robot interaction. The study highlights the effectiveness of these methods, particularly in environments where verbal communication is limited.