Coarse-to-Fine Q-attention with Learned Path Ranking

TL;DR

The paper extends C2F-ARM with Learned Path Ranking (LPR), a module that ranks paths from path planning, Bézier curve sampling, and a learned path policy given a next-best pose. LPR uses a ranking Q-function trained via sparse-reward reinforcement learning to select high-value paths, enabling tasks that require highly specific motions while preserving C2F-ARM’s sample efficiency. Across 16 RLBench tasks and 2 real-world tasks with only 3 demonstrations, the approach maintains performance on simple tasks and improves performance on tasks requiring particular motions such as opening/toilet-seat manipulation, with only modest runtime overhead. This work broadens the practical capabilities of vision-based manipulation under sparse rewards by combining diverse path generators and learned ranking.

Abstract

We propose Learned Path Ranking (LPR), a method that accepts an end-effector goal pose, and learns to rank a set of goal-reaching paths generated from an array of path generating methods, including: path planning, Bezier curve sampling, and a learned policy. The core idea being that each of the path generation modules will be useful in different tasks, or at different stages in a task. When LPR is added as an extension to C2F-ARM, our new system, C2F-ARM+LPR, retains the sample efficiency of its predecessor, while also being able to accomplish a larger set of tasks; in particular, tasks that require very specific motions (e.g. opening toilet seat) that need to be inferred from both demonstrations and exploration data. In addition to benchmarking our approach across 16 RLBench tasks, we also learn real-world tasks, tabula rasa, in 10-15 minutes, with only 3 demonstrations.

Figures (9)

• Figure 1: C2F-ARM+LPR is an extension to C2F-ARM that, when given a next-best pose, learns to rank a set of paths generated from path planning, curve sampling, and a learned path policy. This allows for a larger set of tasks to be learned; in particular, tasks such lifting toilet seat, where only the green path (above) would result in a success.
• Figure 2: In simulation, our method is evaluated on a total of 16 RLBench james2019rlbench tasks. Task Set 1 are tasks that have previously been shown to perform well with C2F-ARM. Task Set 2 are tasks that were chosen due to requiring very particular motion that are necessary for task completion (e.g. opening or closing objects). Note that the position and orientation of objects are placed randomly at the beginning of each episode.
• Figure 3: Learning curves for 8 RLBench tasks. Both methods only receive 10 demos, which are stored in the replay buffer prior to training. Note, the goal here is not to outperform C2F-ARM on these tasks, but to instead show that there is no loss in performance. In Figure \ref{['fig:c2fvspath_new_tasks']} however, the goal is to outperform C2F-ARM, as those tasks require particular motions. Solid lines represent the average evaluation over 5 seeds, while the shaded regions represent the $std$.
• Figure 4: Learning curves for an additional 8 RLBench tasks. Both methods only receive 10 demos, which are stored in the replay buffer prior to training. Unlike Figure \ref{['fig:c2fvspath_orig_tasks']}, the goal here is to outperform C2F-ARM, as these tasks require particular motions; e.g. opening doors, drawers, etc. Solid lines represent the average evaluation over 5 seeds, while the shaded regions represent the $std$.
• Figure 5: Ablation of the learned path policy on a set of 4 RLBench tasks. All methods only receive 10 demos, which are stored in the replay buffer prior to training. Solid lines represent the average evaluation over 5 seeds, while the shaded regions represent the $std$.