Evaluating Gesture Recognition in Virtual Reality
Sandeep Reddy Sabbella, Sara Kaszuba, Francesco Leotta, Pascal Serrarens, Daniele Nardi
TL;DR
The paper tackles data scarcity and benchmarking in gesture recognition for HRI by proposing VR-driven data generation as a scalable complement to real-world data. It outlines a CANOPIES-inspired gesture set and a dual real- and virtual-data collection pipeline, with a three-phase experimentation plan to assess how well models trained on hybrid data transfer to real-world gestures and control a virtual robot. The study highlights the advantages of controlled VR environments for generating diverse, ground-truth gesture datasets, while acknowledging practical costs and the necessity of real-world validation. The work aims to establish data-generation standards and insights into optimal data mixtures for robust gesture recognition in agricultural ground-robot operations and broader HRI contexts.
Abstract
Human-Robot Interaction (HRI) has become increasingly important as robots are being integrated into various aspects of daily life. One key aspect of HRI is gesture recognition, which allows robots to interpret and respond to human gestures in real-time. Gesture recognition plays an important role in non-verbal communication in HRI. To this aim, there is ongoing research on how such non-verbal communication can strengthen verbal communication and improve the system's overall efficiency, thereby enhancing the user experience with the robot. However, several challenges need to be addressed in gesture recognition systems, which include data generation, transferability, scalability, generalizability, standardization, and lack of benchmarking of the gestural systems. In this preliminary paper, we want to address the challenges of data generation using virtual reality simulations and standardization issues by presenting gestures to some commands that can be used as a standard in ground robots.