Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulation

TL;DR

This work tackles open-vocabulary mobile manipulation by introducing Affordance RAG, a zero-shot hierarchical multimodal retrieval framework that uses Affordance-Aware Embodied Memory to ground instructions in pre-explored scene images. The approach combines multi-level semantic representations, region- and object-level affordances, and LLM-guided descriptive retrieval to identify executable target objects and receptacles. It demonstrates state-of-the-art retrieval performance on the WholeHouse-MM benchmark and robust real-world task success (85%), highlighting the importance of affordance-aware reranking and hierarchical memory. The framework paves the way for reliable, instruction-driven manipulation in complex indoor environments without task-specific training.

Abstract

In this study, we address the problem of open-vocabulary mobile manipulation, where a robot is required to carry a wide range of objects to receptacles based on free-form natural language instructions. This task is challenging, as it involves understanding visual semantics and the affordance of manipulation actions. To tackle these challenges, we propose Affordance RAG, a zero-shot hierarchical multimodal retrieval framework that constructs Affordance-Aware Embodied Memory from pre-explored images. The model retrieves candidate targets based on regional and visual semantics and reranks them with affordance scores, allowing the robot to identify manipulation options that are likely to be executable in real-world environments. Our method outperformed existing approaches in retrieval performance for mobile manipulation instruction in large-scale indoor environments. Furthermore, in real-world experiments where the robot performed mobile manipulation in indoor environments based on free-form instructions, the proposed method achieved a task success rate of 85%, outperforming existing methods in both retrieval performance and overall task success.

Figures (6)

• Figure 1: Overview of Affordance RAG for open-vocabulary mobile manipulation. The robot first constructs an embodied memory based on images collected during pre-exploration of the environment. When a free-form instruction is given, hierarchical multimodal retrieval is performed over the embodied memory to identify the target object and receptacle. To improve task success rates, candidates with higher affordance scores are prioritized during retrieval.
• Figure 2: Overview of the Affordance RAG framework. (a) The robot constructs Affordance-Aware Embodied Memory (Affordance Mem) through pre-exploration. Affordance Mem is constructed from three components: Affordance Proposer, Multi-Level Representation, and Area Summarizer. (b) Upon receiving an instruction, hierarchical multimodal retrieval is performed to identify both the target object and the receptacle. This process consists of three stages: Recursive Top-Down Traversal, Multi-Level Fusion, and Affordance-Aware Reranking.
• Figure 3: Comparison of task success rate (SR) on the WholeHouse-MM benchmark. SR@$K$ denotes the percentage of samples in which both the target object and the receptacle are correctly retrieved within the top-$K$ results.
• Figure 4: Qualitative results on the WholeHouse-MM benchmark. The given $\bm{x}_\mathrm{inst}$ was "Take a photo frame from the side table in the bedroom and place it on the dining table with a bouquet of flowers." (a) Target object and (b) receptacle: Top-2 retrieved images are shown for both the our method and the best baseline method (BEiT-3 beit3). The ground-truth image is highlighted with a green border.
• Figure 5: Sensitivity analysis of the weight $\alpha$ in Multi-Level Fusion. The hyperparameter $\alpha$ balances the contribution between regional semantics and visual semantics.