Optimistic Reinforcement Learning-Based Skill Insertions for Task and Motion Planning

TL;DR

This work addresses long-horizon manipulation where action effects are uncertain by integrating reinforcement-learning skills into a symbolic Task and Motion Planning (TAMP) framework. It introduces neuro-symbolic RL skills encoded as $\phi=\langle\bar{v},\omega,\pi,\Theta,\sigma\rangle$ with optimistic substitutions and plan refinement via Observe actions, enabled by a state discriminator and a sub-goal generator. Empirical results across four probabilistic domains and real-world tests with a Franka robot show improved planning efficiency and execution success compared to heuristic, plain-sampling, and synergistic RL baselines. By grounding probabilistic actions in learned policies while maintaining symbolic grounding, the approach extends TAMP capabilities to uncertain actions with practical impact for robust robotic manipulation.

Abstract

Task and motion planning (TAMP) for robotics manipulation necessitates long-horizon reasoning involving versatile actions and skills. While deterministic actions can be crafted by sampling or optimizing with certain constraints, planning actions with uncertainty, i.e., probabilistic actions, remains a challenge for TAMP. On the contrary, Reinforcement Learning (RL) excels in acquiring versatile, yet short-horizon, manipulation skills that are robust with uncertainties. In this letter, we design a method that integrates RL skills into TAMP pipelines. Besides the policy, a RL skill is defined with data-driven logical components that enable the skill to be deployed by symbolic planning. A plan refinement sub-routine is designed to further tackle the inevitable effect uncertainties. In the experiments, we compare our method with baseline hierarchical planning from both TAMP and RL fields and illustrate the strength of the method. The results show that by embedding RL skills, we extend the capability of TAMP to domains with probabilistic skills, and improve the planning efficiency compared to the previous methods.

Figures (14)

• Figure 1: A table rearranging task. In this scenario, the goal is to deliver a plate and a cup on the table to another. Besides deterministic actions such as pick and place (b, c, e, f), it requires several challenging skills to achieve such a task such as (a) pushing the plate to the edge of the table in order to obtain grasping space and (d) retrieving the cup back to the workspace with the bar.
• Figure 2: The structure of a RL skill in TAMP. The blue pipeline indicates the task planning while the green pipeline is the motion planning, they are mediated by a middle layer indicated by the turquoise entities. Our RL probabilistic skills contain a RL policy in the motion level, a data-driven state discriminator, and a sub-goal generator in the middle layer. The RL probabilistic skills can connect with an operator by generating symbolic preconditions and effects. Therefore, the orange dash line frames out a RL skill. On the right side, the output from the state discriminator and sub-goal generator are shown. The state discriminator is a binary classifier network that can discriminate the valid initial states (in green) and invalid initial states (in red) for a skill. The sub-goal generator provides reachable sub-goals (in green) according to the initial state and the used RL policy, while unreachable sub-goals (in red) will be filtered out. The dash lines around these states represent the uncertainty boundaries of the predicted final state since the outcome of probabilistic actions, like edgepush or retrieve, cannot be determined beforehand.
• Figure 3: The EdgePush training environment consists of different states in the simulation presented in the first column. The second column represents the observations, i.e., the encoded depth images. According to the depth sensor, the object, table edge, and ground are encoded with different numbers. The third column illustrates that rewards will be given by sampling possible grasps. The red end-effectors indicate colliding grasping poses, while the green ones indicate collision-free grasping poses.
• Figure 4: The data generation for state discriminator and sub-goal generator. After training, we run episodes with the trained policy. After each episode, we compare the actual final effect state and the initial sub-goal, if they are close enough then we save the initial state and the sub-goal in this episode. the black lines indicate the policy-environment interaction loop and the blue lines and shades indicate the data selection.
• Figure 5: State discriminator visualization. (a) The heatmap illustrates the predicted success possibility from the (partial) initial state $x_0$. The axes are the spatial positions. The yellow line, where $p(\texttt{s}=1)=0.5$ is the boundary of our binary classifier. (b) We test the state discriminator by randomly sampling initial states of cups, the green cups indicate the states are valid while the red cups are invalid states.