A Task-Efficient Reinforcement Learning Task-Motion Planner for Safe Human-Robot Cooperation

TL;DR

The paper addresses safety-efficiency trade-offs in human-robot collaboration under dynamic human motion. It proposes a hybrid framework combining a re-RRT* motion planner with a PPO-trained RL task planner to propose safe, efficient task goals while the motion planner maintains collision-free execution. A reward function that balances task success, collision avoidance, replanning time, and proximity guides learning, yielding fewer replans and task failures, explicitly: $r= \alpha_{0}\cdot [\text{task\_achieved}] - \alpha_{1}\cdot [\text{collision}] - \alpha_{2}\cdot c_k - \alpha_{3}\cdot [d(\mathbf{p}_{ee}, \mathbf{p}_{\tau_k})]$. Experiments in simulation and on a Franka Panda demonstrate improved safety and task efficiency, with real-world perception via ZED2 and ArUco integrated through ROS/Gazebo.

Abstract

In a Human-Robot Cooperation (HRC) environment, safety and efficiency are the two core properties to evaluate robot performance. However, safety mechanisms usually hinder task efficiency since human intervention will cause backup motions and goal failures of the robot. Frequent motion replanning will increase the computational load and the chance of failure. In this paper, we present a hybrid Reinforcement Learning (RL) planning framework which is comprised of an interactive motion planner and a RL task planner. The RL task planner attempts to choose statistically safe and efficient task sequences based on the feedback from the motion planner, while the motion planner keeps the task execution process collision-free by detecting human arm motions and deploying new paths when the previous path is not valid anymore. Intuitively, the RL agent will learn to avoid dangerous tasks, while the motion planner ensures that the chosen tasks are safe. The proposed framework is validated on the cobot in both simulation and the real world, we compare the planner with hard-coded task motion planning methods. The results show that our planning framework can 1) react to uncertain human motions at both joint and task levels; 2) reduce the times of repeating failed goal commands; 3) reduce the total number of replanning requests.

Figures (11)

• Figure 1: Human-Robot Cooperation (HRC) environment where the human and the robot share a workspace. The robot and the human share the same manipulation goal which is to pick up and pack all the objects on the table. On top of the motion planning in the joint space to avoid the human arms, the robot also needs to avoid the occupation in the task space where the human has higher possibility working in, which is depicted as the red zone.
• Figure 2: The planning framework is comprised of a task planner and a motion planner. The RL task planner gives task commands to the motion planner, the motion planner attempts to refine task commands by replanning safe trajectories. Combining the number of replanning requests and weather the task is accessible, the motion planner evaluates the task planner's decisions by giving rewards, which also helps the RL agent to improve the ability to make better decisions.
• Figure 3: The tradeoff between the planning frequency and optimality. (a) The end-effector trajectory elongates when the replanning frequency increases, because following frequently-changing sub-optimal trajectories will make the end-effector constantly change moving directions. (b) The planning failure ratio also increases when the replanning frequency increases, because of the lack of time for the planner to find the solutions. On the other hand, reducing the replanning frequency will cause the severe motion delay, which will deteriorate the safety.
• Figure 4: The effect of the re-RRT* motion planner. The blue line and red line represent the original trajectory, however, the human arms change position unexpectedly, the previous trajectory is not valid anymore after the blue section, a new successive trajectory (green) is planned to cope with the change.
• Figure 5: The observation feature matrix visualization. (a) The task objects and human arms in the simulation, green boxes are the task objects and the blue cylinders-spheres objects indicate the human arms. (b) The visualization of the integer-encoding observed feature matrix, which is a $10\times10$, three-layer matrix. The green color indicates 1 in the matrix elements, the blue color indicates -1 in the matrix elements, while the white color indicates 0.