Communicating human intent to a robotic companion by multi-type gesture sentences

TL;DR

A gesture pseudo-language is proposed and it is shown how multiple types of gestures can be combined to express human intent to a robot, lowering the execution time by up to 60%, compared to direct teleoperation.

Abstract

Human-Robot collaboration in home and industrial workspaces is on the rise. However, the communication between robots and humans is a bottleneck. Although people use a combination of different types of gestures to complement speech, only a few robotic systems utilize gestures for communication. In this paper, we propose a gesture pseudo-language and show how multiple types of gestures can be combined to express human intent to a robot (i.e., expressing both the desired action and its parameters - e.g., pointing to an object and showing that the object should be emptied into a bowl). The demonstrated gestures and the perceived table-top scene (object poses detected by CosyPose) are processed in real-time) to extract the human's intent. We utilize behavior trees to generate reactive robot behavior that handles various possible states of the world (e.g., a drawer has to be opened before an object is placed into it) and recovers from errors (e.g., when the scene changes). Furthermore, our system enables switching between direct teleoperation of the end-effector and high-level operation using the proposed gesture sentences. The system is evaluated on increasingly complex tasks using a real 7-DoF Franka Emika Panda manipulator. Controlling the robot via action gestures lowered the execution time by up to 60%, compared to direct teleoperation.

Figures (9)

• Figure 1: Human operating a robot with gestures.
• Figure 2: Metric gesture example, they are auxiliary and optional. Pinch distance (left), point direction (center), hands distance (right)
• Figure 3: Behavior tree structure example is solved as completing object state preconditions. For example, picking up an object stacked up with other objects requires to first unstacking all objects above it.
• Figure 4: The gesture detection output of a single episode. The upper plot shows the likelihood of static (left legend) and dynamic (right legend) gestures. The bottom plot shows the visibility of the hands and the activation of the gestures. First was detected the static gesture point followed by grab and dynamic gesture swipe down.
• Figure 5: System diagram. $\mathcal{G}$ represents gesture classifier, $\mathcal{M}$ is mapping from gestures to human intent, $\mathcal{A}$ is robotic action generation using Behaviour tree, $\mathcal{D}$ is Deictic gesture execution. Blue blocks represent user inputs, and yellow blocks are robot output modes.