Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Exploration

TL;DR

This work proposes Long-term Memory Embodied Exploration (LMEE), a novel method that fine-tunes a multimodal large language model through reinforcement learning to encourage active memory querying and achieves proactive exploration.

Abstract

An ideal embodied agent should possess lifelong learning capabilities to handle long-horizon and complex tasks, enabling continuous operation in general environments. This not only requires the agent to accurately accomplish given tasks but also to leverage long-term episodic memory to optimize decision-making. However, existing mainstream one-shot embodied tasks primarily focus on task completion results, neglecting the crucial process of exploration and memory utilization. To address this, we propose Long-term Memory Embodied Exploration (LMEE), which aims to unify the agent's exploratory cognition and decision-making behaviors to promote lifelong learning.We further construct a corresponding dataset and benchmark, LMEE-Bench, incorporating multi-goal navigation and memory-based question answering to comprehensively evaluate both the process and outcome of embodied exploration. To enhance the agent's memory recall and proactive exploration capabilities, we propose MemoryExplorer, a novel method that fine-tunes a multimodal large language model through reinforcement learning to encourage active memory querying. By incorporating a multi-task reward function that includes action prediction, frontier selection, and question answering, our model achieves proactive exploration. Extensive experiments against state-of-the-art embodied exploration models demonstrate that our approach achieves significant advantages in long-horizon embodied tasks.

Figures (18)

• Figure 1: We propose Long-term Memory Embodied Exploration, which aims to collect episodic memories during Multi-goal Navigation and introduces Memory-based Question Answering to unify and evaluate the model’s cognitive and decision-making abilities.
• Figure 2: The construction process of Long-term Memory Embodied Exploration and data statistics.
• Figure 3: Illustration of training in MemoryExplorer. Given a task instruction, the multi-view observations, and a goal-oriented question. Model retrieves relevant multimodal memories from the episodic memory bank using tools, analyzes the current information alongside the retrieved memories to understand the progress of the long-term task, and performs ACTION prediction, FRONTIER selection, and question ANSWER. The policy model output response calculates the reward using a Multi-Task Reward function and is fine-tuned using GRPO.
• Figure 4: Qualitative example of LMEE-Bench.
• Figure 5: Training reward curve and tool usage percentage.