Preemptive Motion Planning for Human-to-Robot Indirect Placement Handovers

TL;DR

A novel prediction-planning pipeline is introduced that allows the robot to preemptively move towards the human agent's intended placement location using gaze and gestures as model inputs as well as the practical benefits of using such a pipeline through a human-robot case study.

Abstract

As technology advances, the need for safe, efficient, and collaborative human-robot-teams has become increasingly important. One of the most fundamental collaborative tasks in any setting is the object handover. Human-to-robot handovers can take either of two approaches: (1) direct hand-to-hand or (2) indirect hand-to-placement-to-pick-up. The latter approach ensures minimal contact between the human and robot but can also result in increased idle time due to having to wait for the object to first be placed down on a surface. To minimize such idle time, the robot must preemptively predict the human intent of where the object will be placed. Furthermore, for the robot to preemptively act in any sort of productive manner, predictions and motion planning must occur in real-time. We introduce a novel prediction-planning pipeline that allows the robot to preemptively move towards the human agent's intended placement location using gaze and gestures as model inputs. In this paper, we investigate the performance and drawbacks of our early intent predictor-planner as well as the practical benefits of using such a pipeline through a human-robot case study.

Figures (6)

• Figure 1: Robot system pipeline. (a) shows the high level modules of the system which are color coded. (b) shows the individual components making up each of the high level modules. Predictive and definitive actions are defined in Sec. \ref{['subsec:motion_planning']}.
• Figure 2: An illustration of the intent model architecture. As shown, the model is a recurrent convolutional network. Recurrent units act as the encoder, where the pose & gaze feature vector is fed into the gated recurrent units (GRU) at every time step. Here, the hidden states fed back into the GRUs represent the encoded state of past time steps. Convolutional layers act as the decoder which perform a series of transpose convolutions from a 1-D encoded state vector. The dimensions of the convolutional layers are shown for a heatmap output of size $5 \times 10$, which is the size used in the case study.
• Figure 3: An example prediction sequence. The red vector located on the participant's head is the detected face norm. The red dot indicates the gaze and table plane intersection point $\boldsymbol \psi$ discussed in Sec. \ref{['subsec:features']}. The green skeleton shows the detected shoulder, elbow, and palm positions. For each frame (a-h) along the trajectory, the corresponding heatmap can be seen underneath. Note that at the start of the trajectory, the model is unsure of the human's placement intent. As more and more of trajectory is realized, a prediction with increasing confidence can be seen being made.
• Figure 4: Example sequence with preemptive motion planning. The robot is in the ready position in frame (a). As more and more of the trajectory is observed, the robot correctly predicts the human's placement intent and is able to move to a position to pick up the object as shown in frame (f).
• Figure 5: The first row and second row are the response and start-to-grab time boxplots, respectively. Each column corresponds to one of the eleven grid locations used in the experiment. Each subplot consists of two boxplots labeled R for reactive and P for preemptive. The yellow line corresponds to the median while the blue dot is the mean. As shown, there is a clear decrease in both response and start-to-grab times when employing preemptive control.