Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigation

TL;DR

This work tackles long-horizon embodied ObjectNav by integrating a global memory module with egocentric perception. The Mem2Ego framework adaptively retrieves task-relevant cues from a frontier, landmark, and visitation memory and fuses them with panoramic egocentric observations to guide VLM-driven decision making. Empirical results on Habitat 3.0 across HSSD datasets show superior SR and SPL over state-of-the-art baselines, and a supervised fine-tuning pipeline for a open-source Llama model further boosts performance beyond GPT-4o. The approach demonstrates that aligning global contextual information with local perception can significantly enhance spatial reasoning and navigation efficiency in complex environments.

Abstract

Recent advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs) have made them powerful tools in embodied navigation, enabling agents to leverage commonsense and spatial reasoning for efficient exploration in unfamiliar environments. Existing LLM-based approaches convert global memory, such as semantic or topological maps, into language descriptions to guide navigation. While this improves efficiency and reduces redundant exploration, the loss of geometric information in language-based representations hinders spatial reasoning, especially in intricate environments. To address this, VLM-based approaches directly process ego-centric visual inputs to select optimal directions for exploration. However, relying solely on a first-person perspective makes navigation a partially observed decision-making problem, leading to suboptimal decisions in complex environments. In this paper, we present a novel vision-language model (VLM)-based navigation framework that addresses these challenges by adaptively retrieving task-relevant cues from a global memory module and integrating them with the agent's egocentric observations. By dynamically aligning global contextual information with local perception, our approach enhances spatial reasoning and decision-making in long-horizon tasks. Experimental results demonstrate that the proposed method surpasses previous state-of-the-art approaches in object navigation tasks, providing a more effective and scalable solution for embodied navigation.

Figures (7)

• Figure 1: Workflow of our proposed method. Our method maintains three types of memories and project cues from them onto the egocentric images for goal location prediction. Further details are provided in Section \ref{['sec:method']}
• Figure 2: Pipeline of SFT data collection. The ground-truth trajectory and floor edge are used to extract target marker and candidate markers, respectively. Marker description and selection data is generated for model fine-tuning.
• Figure 3: Reduction of steps per episode on the HSSD-Hard dataset. For each episode, we compute the difference in the number of steps required between our method and the designated baseline. A positive value (blue color) indicates a reduction in steps when using our method, while a negative value (orange color) signifies an increase. Episode IDs are sorted based on these delta values.
• Figure 4: Demonstration of memory-augmented decision making process from a real HSSD episode. The full prompts and GPT-4o response are provided in Appendix \ref{['sec:app_case_study']}
• Figure 5: Results for different maximum steps. Experiments are conducted on the HSSD dataset. While SFT Llama3.2-11B consistently achieves the best performance, its advantage over the other two models is most pronounced at 300 maximum steps.