ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation

TL;DR

ReLMoGen addresses long-horizon mobile manipulation by lifting the action space from low-level motor commands to subgoals for a motion generator. It combines a learned Subgoal Generation Policy with a classical motion generator to plan and execute trajectories between subgoals, forming a lifted MDP that improves exploration and task success. Across seven diverse tasks in photo-realistic simulations, ReLMoGen outperforms strong RL and HRL baselines and shows robust transfer across motion planners, implying strong practical potential for real robots. The framework convincingly demonstrates how learning can effectively steer planning-based control, enabling efficient, transferable, and interpretable policies for complex robotic manipulation.

Abstract

Many Reinforcement Learning (RL) approaches use joint control signals (positions, velocities, torques) as action space for continuous control tasks. We propose to lift the action space to a higher level in the form of subgoals for a motion generator (a combination of motion planner and trajectory executor). We argue that, by lifting the action space and by leveraging sampling-based motion planners, we can efficiently use RL to solve complex, long-horizon tasks that could not be solved with existing RL methods in the original action space. We propose ReLMoGen -- a framework that combines a learned policy to predict subgoals and a motion generator to plan and execute the motion needed to reach these subgoals. To validate our method, we apply ReLMoGen to two types of tasks: 1) Interactive Navigation tasks, navigation problems where interactions with the environment are required to reach the destination, and 2) Mobile Manipulation tasks, manipulation tasks that require moving the robot base. These problems are challenging because they are usually long-horizon, hard to explore during training, and comprise alternating phases of navigation and interaction. Our method is benchmarked on a diverse set of seven robotics tasks in photo-realistic simulation environments. In all settings, ReLMoGen outperforms state-of-the-art Reinforcement Learning and Hierarchical Reinforcement Learning baselines. ReLMoGen also shows outstanding transferability between different motion generators at test time, indicating a great potential to transfer to real robots.

Figures (10)

• Figure 1: (top) We propose to integrate motion generation into a reinforcement learning loop to lift the action space from low-level robot actions $a$ to subgoals for the motion generator $a'$ (bottom) The mobile manipulation tasks we can solve with ReLMoGen are composed by a sequence of base and arm subgoals (e.g. pushing open a door for Interactive Navigation).
• Figure 2: Two types of action parameterization of ReLMoGen and network architecture of SGP-D and SGP-R.
• Figure 3: The simulation environments and tasks. (a)(b) navigation-only and manipulation-only tasks, (c)(d) three Interactive Navigation tasks, (e)(f) two Mobile Manipulation tasks.
• Figure 4: Training curves for ReLMoGen and the baselines (SAC, OAC, and HRL4IN). ReLMoGen achieves higher reward with the same number of environment episodes and higher task completion for all seven tasks while the baselines often converge to sub-optimal solutions. The curve indicates the mean and standard deviation of the return across three random seeds. Note that the x-axis indicates environment episodes rather than steps to allow for a fair comparison between solutions that use actions with different time horizons.
• Figure 5: Exploration of ReLMoGen-R and SAC. (a) shows the 2D projection of latent state space: SAC traverses nearby states with low-level actions, while ReLMoGen-R jumps between distant states linked by a motion plan. (b) shows the physical locations visited by ReLMoGen-R and SAC in 100 episodes: ReLMoGen-R covers a much larger area. (c) shows a top-down map of meaningful interactions (duration $\ge$1s) during exploration. ReLMoGen-R is able to interact with the environment more than SAC.