Spatially Generalizable Mobile Manipulation via Adaptive Experience Selection and Dynamic Imagination

TL;DR

The paper tackles long-horizon mobile manipulation by addressing two key issues: sample efficiency and spatial generalization. It introduces Adaptive Experience Selection AES to prioritize informative episodic fragments and a Recurrent State-Space Model RSSM for model-based dynamic imagination used in Model-Predictive Forward Planning MPFP. Together, AES memorizes task-relevant experiences while RSSM imagines future state transitions to guide planning, enabling robust skill chaining and spatial generalization across layouts. The approach, demonstrated in an EE-centric MM framework on real hardware, shows superior performance and generalization over strong baselines and validated sim-to-real feasibility.

Abstract

Mobile Manipulation (MM) involves long-horizon decision-making over multi-stage compositions of heterogeneous skills, such as navigation and picking up objects. Despite recent progress, existing MM methods still face two key limitations: (i) low sample efficiency, due to ineffective use of redundant data generated during long-term MM interactions; and (ii) poor spatial generalization, as policies trained on specific tasks struggle to transfer to new spatial layouts without additional training. In this paper, we address these challenges through Adaptive Experience Selection (AES) and model-based dynamic imagination. In particular, AES makes MM agents pay more attention to critical experience fragments in long trajectories that affect task success, improving skill chain learning and mitigating skill forgetting. Based on AES, a Recurrent State-Space Model (RSSM) is introduced for Model-Predictive Forward Planning (MPFP) by capturing the coupled dynamics between the mobile base and the manipulator and imagining the dynamics of future manipulations. RSSM-based MPFP can reinforce MM skill learning on the current task while enabling effective generalization to new spatial layouts. Comparative studies across different experimental configurations demonstrate that our method significantly outperforms existing MM policies. Real-world experiments further validate the feasibility and practicality of our method.

Figures (4)

• Figure 1: (a) Our AES enables RSSM to pay more attention to critical episodic memories in experience trajectories, which are used to improve and spatially generalize RL-based MM agents. (b) RSSM cooperates with RL-based MM skills for $H$-horizon dynamic imagination to achieve MPFP. The most promising elite path is selected for execution. (c) An example of MM tasks that consist of heterogeneous skills, i.e., navigation $\rightarrow$ picking $\rightarrow$ navigation $\rightarrow$ placing.
• Figure 2: (a) An illustration of the structure, unfolding, and training process of RSSM. (b) An illustration of which key experience fragments the AES focuses on, including the experience of interacting with objects, opening doors, and passing through narrow areas. These experience segments are likely to have collisions and IK solving failures that affect task success. (c) An illustration of RSSM-based MPFP.
• Figure 3: We consider four different experimental configurations: (a) cross-room, (b) warehouse-oriented, (c) home-scene-oriented, and (d) dynamic-scene-oriented (white dynamic obstacles) mobile manipulations. (e) An illustration of the cross-scene MM task. In each episode, the positions of the robot, the obstacles, the desks, the picking point (Goal$_1$), the start point of opening the door (Goal$_2$), the end point of opening the door (Goal$_3$), and the placing point (Goal$_4$) are randomly initialized. This task requires the robot to pick up an object and then open a door to another room to place the object in a specified position in the room. More details can be found in the supplementary material.
• Figure 4: TCR metrics change with the number of training episodes in the cross-room experimental configuration.