Next-Future: Sample-Efficient Policy Learning for Robotic-Arm Tasks

TL;DR

The paper tackles the inefficiency of deep reinforcement learning for robotic-arm manipulation in multi-goal settings with sparse binary rewards. It introduces Next-Future, a principled single-step goal relabeling strategy that guarantees non-negative rewards for the first of multiple replays by setting the next achieved state as the virtual goal, while using remaining relabelings from future states to propagate values. To stabilize learning and reduce overestimation, it employs a distributional critic with truncated quantile critics (TQC) and an ensemble of heads, improving value approximation and policy updates. Across eight simulated robotic-arm tasks and ten seeds, Next-Future yields substantial gains in sample efficiency (seven tasks) and higher maximal success rates (six tasks), with real-world experiments confirming practical feasibility. The approach also demonstrates compatibility with existing HER extensions (EBP, CHER), offering a flexible and scalable pathway to more accurate, data-efficient multi-goal DRL for robotics.

Abstract

Hindsight Experience Replay (HER) is widely regarded as the state-of-the-art algorithm for achieving sample-efficient multi-goal reinforcement learning (RL) in robotic manipulation tasks with binary rewards. HER facilitates learning from failed attempts by replaying trajectories with redefined goals. However, it relies on a heuristic-based replay method that lacks a principled framework. To address this limitation, we introduce a novel replay strategy, "Next-Future", which focuses on rewarding single-step transitions. This approach significantly enhances sample efficiency and accuracy in learning multi-goal Markov decision processes (MDPs), particularly under stringent accuracy requirements -- a critical aspect for performing complex and precise robotic-arm tasks. We demonstrate the efficacy of our method by highlighting how single-step learning enables improved value approximation within the multi-goal RL framework. The performance of the proposed replay strategy is evaluated across eight challenging robotic manipulation tasks, using ten random seeds for training. Our results indicate substantial improvements in sample efficiency for seven out of eight tasks and higher success rates in six tasks. Furthermore, real-world experiments validate the practical feasibility of the learned policies, demonstrating the potential of "Next-Future" in solving complex robotic-arm tasks.

Figures (9)

• Figure 1: The robotic arm perform the task of push a cube from an initial position to specified final position. Left column: shows simulation environment. Right column: shows corresponding output on the real robotic arm platform.
• Figure 2: Value function illustration of an MDP for multi-goal RL. Three horizontal slices taken from this multi-goal MDP represent separate standard MDPs with their unique goals. While the state-space is in 2D (x and y coordinates) the goal space is only 1D and matches the y-coordinate of the agent. It is observed visually that goal-conditioning the value function alters the underlying MDP differently across the goal space.
• Figure 3: Illustration of goal-conditioned trajectories where transitions are color-coded with their respective binary rewards (red for negative and green for non-negative reward). a) All transitions in the original trajectory receive a negative reward since the goal state is more than $\epsilon_{R}$ away from each state. This makes learning difficult for standard RL algorithms. b) If each transition is replayed where the final state of the environment is considered as the new goal then those nearby it will be associated with a non-negative reward and learning will be facilitated. This corresponds to Final strategy of HER work andrychowicz2017hindsight. c) Reducing $\epsilon_{R}$ to improve the accuracy of the policy eliminates most of the rewards and HER's performance degrades.
• Figure 4: Illustration of state transitions in a multi-goal MDP and their corresponding rewards under different goal selection strategies (red for negative and green for non-negative reward). a) and b): All transitions of the original trajectory receive negative rewards because no transition has achieved the goal of attaining zero y-coordinate. c) and d): Applying Final goal selection strategy lifts the original trajectory up in the goal axis to the last achieved state of the episode. Consequently, last transition is awarded non-negatively. An intermediate transition is also given a non-negative reward as it happens to achieve the same goal state after the augmentation. e) and f) strategy randomly moves each state vertically in the goal coordinate and there is no certainty about how the rewards will change. On the other hand, Next strategy deterministicly converts each transition to a successful one by altering the goal state as the next achieved state.
• Figure 5: Experimental setup in simulation for the popular and newly introduced robotic-arm tasks, respectively.