Dynamic Hand Gesture-Featured Human Motor Adaptation in Tool Delivery using Voice Recognition

TL;DR

An innovative human-robot collaborative framework that seamlessly integrates hand gesture and dynamic movement recognition, voice recognition, and a switchable control adaptation strategy is introduced that facilitates higher efficiency tool delivery, without significantly distracting from human intents.

Abstract

Human-robot collaboration has benefited users with higher efficiency towards interactive tasks. Nevertheless, most collaborative schemes rely on complicated human-machine interfaces, which might lack the requisite intuitiveness compared with natural limb control. We also expect to understand human intent with low training data requirements. In response to these challenges, this paper introduces an innovative human-robot collaborative framework that seamlessly integrates hand gesture and dynamic movement recognition, voice recognition, and a switchable control adaptation strategy. These modules provide a user-friendly approach that enables the robot to deliver the tools as per user need, especially when the user is working with both hands. Therefore, users can focus on their task execution without additional training in the use of human-machine interfaces, while the robot interprets their intuitive gestures. The proposed multimodal interaction framework is executed in the UR5e robot platform equipped with a RealSense D435i camera, and the effectiveness is assessed through a soldering circuit board task. The experiment results have demonstrated superior performance in hand gesture recognition, where the static hand gesture recognition module achieves an accuracy of 94.3\%, while the dynamic motion recognition module reaches 97.6\% accuracy. Compared with human solo manipulation, the proposed approach facilitates higher efficiency tool delivery, without significantly distracting from human intents.

Figures (6)

• Figure 1: A practical validation platform designed to assess multimodal interaction during the electrical circuit repair handover task.
• Figure 2: Schematic representation of the comprehensive framework for HRC in dynamic tool delivery. The framework encompasses two fundamental stages: robot fetching and tool delivery. In the robot-fetching stage, voice command recognition enables users to specify desired tools verbally. The robot employs visual feedback to recognize and track the user's hand, estimating its 3D pose and discerning the user's intention. In the tool-delivering stage, real-time hand pose estimation through a depth camera ensures precise tool delivery.
• Figure 3: Sample results from gesture and hand movement recognition frames, illustrating various scenarios. To enhance clarity, depth and RGB images have been combined, with pixels corresponding to point cloud data beyond the defined range omitted. (a) No hand. (b) Open hand gesture. (c) Closed hand gesture. (d) Occupied hand gesture. (e) Low urgency hand movement. (f) Medium-urgency hand movement. (g) High-urgency hand movement. (h) 'Go away' hand movement.
• Figure 4: (a) Dual-camera images captured by the head-mounted PupilLabs Core eye-tracker. (b) Temporal evolution of gaze positions. (c) Heatmap representing gaze distribution based on eye tracking data.
• Figure 5: Robot positional error relative to the target position over time: (a) LQR control with the state matrix to all zeros and the control input matrix as an identity matrix. (b) PID control with proportional gains set to 0.1, integral gains set to 0, and derivative gains set to 0.2 for all x, y, and z axes.