Interpretable Multimodal Gesture Recognition for Drone and Mobile Robot Teleoperation via Log-Likelihood Ratio Fusion

TL;DR

A multimodal gesture recognition framework that integrates inertial data from Apple Watches on both wrists with capacitive sensing signals from custom gloves is proposed and a late fusion strategy based on the log-likelihood ratio (LLR) is designed, which not only enhances recognition performance but also provides interpretability by quantifying modality-specific contributions.

Abstract

Human operators are still frequently exposed to hazardous environments such as disaster zones and industrial facilities, where intuitive and reliable teleoperation of mobile robots and Unmanned Aerial Vehicles (UAVs) is essential. In this context, hands-free teleoperation enhances operator mobility and situational awareness, thereby improving safety in hazardous environments. While vision-based gesture recognition has been explored as one method for hands-free teleoperation, its performance often deteriorates under occlusions, lighting variations, and cluttered backgrounds, limiting its applicability in real-world operations. To overcome these limitations, we propose a multimodal gesture recognition framework that integrates inertial data (accelerometer, gyroscope, and orientation) from Apple Watches on both wrists with capacitive sensing signals from custom gloves. We design a late fusion strategy based on the log-likelihood ratio (LLR), which not only enhances recognition performance but also provides interpretability by quantifying modality-specific contributions. To support this research, we introduce a new dataset of 20 distinct gestures inspired by aircraft marshalling signals, comprising synchronized RGB video, IMU, and capacitive sensor data. Experimental results demonstrate that our framework achieves performance comparable to a state-of-the-art vision-based baseline while significantly reducing computational cost, model size, and training time, making it well suited for real-time robot control. We therefore underscore the potential of sensor-based multimodal fusion as a robust and interpretable solution for gesture-driven mobile robot and drone teleoperation.

Figures (7)

• Figure 1: An example of drone control using the proposed sensor-based gesture recognition.
• Figure 2: Hardware setup with textile sensing gloves and Apple Watch devices on both hands is shown in \ref{['fig:all_devices']}. Purple outlines mark capacitive sensors (bottom) and IMU sensors (top), while orange outlines mark Apple Watch devices on the wrist used for IMU sensing and quaternion orientation.
• Figure 3: Framework architecture for sensor-based gesture recognition on our dataset. Each sensor modality is processed individually through convolutional and temporal feature extraction layers, and the resulting modality-specific features are fused using either log-likelihood ratio (LLR) fusion \ref{['eq:llr_fusion']} or self-attention fusion for classification.
• Figure 4: Example gesture classes are illustrated, including 'brake,' 'brake fire right,' and 'move away.' Each gesture is represented by two sequential image frames, displaying a non-mirrored, first-person perspective.
• Figure 5: Computational resource comparison between sensor-based (orange) and vision-based (blue) models. From left to right, the plots compare GFLOPs per inference, model size, and training time. Training time is reported as the average across all dataset split variations.