Context-aware robot control using gesture episodes

TL;DR

The paper addresses intuitive robot control in HRI by inferring user intent from gesture episodes within a contextual scene. It proposes a three-stage pipeline combining real-time gesture recognition (static and dynamic gestures), context-aware intent mapping using a probabilistic neural network, and a behavior-tree-based generation of robotic actions. Key contributions include a context-dependent gesture framework that supports gesture combinations, a synthetic data generation approach with varying context dependency, and demonstrations of robustness and autonomy in simulation. The approach advances gesture interfaces beyond one-to-one mappings, enabling personalized and robust robot control in partially automated manufacturing settings, with open-source code and data.

Abstract

Collaborative robots became a popular tool for increasing productivity in partly automated manufacturing plants. Intuitive robot teaching methods are required to quickly and flexibly adapt the robot programs to new tasks. Gestures have an essential role in human communication. However, in human-robot-interaction scenarios, gesture-based user interfaces are so far used rarely, and if they employ a one-to-one mapping of gestures to robot control variables. In this paper, we propose a method that infers the user's intent based on gesture episodes, the context of the situation, and common sense. The approach is evaluated in a simulated table-top manipulation setting. We conduct deterministic experiments with simulated users and show that the system can even handle personal preferences of each user.

Figures (8)

• Figure 1: The diagram of the proposed system shows the whole pipeline from hand observations to robotic actions.
• Figure 2: Graphical model of Probabilistic neural network training for the mapping task $\mathcal{M}$ (Gestures + Context $\rightarrow$ Intent), $D$ is # of dataset samples, $G$ is # of gestures, $C$ context observations, $I$ is user intent probabilities, and $N$ is # of classification layers. Normal and Categorical tags represent type of probability distribution. Categorical type represents discrete probabilistic distribution (see Sec.: \ref{['sec:nn_method']}).
• Figure 3: Construction of action sequence $\textbf{a}$ from intent $\textbf{i}$ using a Behavior tree approach.
• Figure 4: Experimental setup. On the left side is Leap Motion Sensor Weichert_Bachmann_Rudak_Fisseler_2013 for hand tracking. On the right side is a Robot setup with Panda Manipulator with an initialized scene. The yellow dots show the robot scene grid.
• Figure 5: Random scene example generation.