Lifelong Embodied Navigation Learning

TL;DR

Uni-Walker is proposed, a lifelong embodied navigation framework that decouples navigation knowledge into task-shared and task-specific components with Decoder Extension LoRA (DE-LoRA) and demonstrates the superiority of Uni-Walker for building universal navigation agents with lifelong learning.

Abstract

Embodied navigation agents powered by large language models have shown strong performance on individual tasks but struggle to continually acquire new navigation skills, which suffer from catastrophic forgetting. We formalize this challenge as lifelong embodied navigation learning (LENL), where an agent is required to adapt to a sequence of navigation tasks spanning multiple scenes and diverse user instruction styles, while retaining previously learned knowledge. To tackle this problem, we propose Uni-Walker, a lifelong embodied navigation framework that decouples navigation knowledge into task-shared and task-specific components with Decoder Extension LoRA (DE-LoRA). To learn the shared knowledge, we design a knowledge inheritance strategy and an experts co-activation strategy to facilitate shared knowledge transfer and refinement across multiple navigation tasks. To learn the specific knowledge, we propose an expert subspace orthogonality constraint together and a navigation-specific chain-of-thought reasoning mechanism to capture specific knowledge and enhance instruction-style understanding. Extensive experiments demonstrate the superiority of Uni-Walker for building universal navigation agents with lifelong learning.

Figures (7)

• Figure 1: Illustration of the proposed lifelong embodied navigation learning (LENL) task. The proposed Uni-Walker is able to evolve and continually learn multiple new navigation tasks based on learned navigation knowledge, for developing universal embodied navigation. In lifelong learning, the sequential navigation tasks include multi-scenes and multi-instruction styles (VLN, OLN, DUN).
• Figure 2: Illustration of the LENL performance. (a) The catastrophic forgetting phenomenon under the LENL settings. (b) Our proposed Uni-Walker has a better anti-forgetting performance.
• Figure 3: Illustration of the proposed Uni-Walker pipeline. It includes (a) a Decoder Extension LoRA Adaptation to achieve progressive knowledge decoupled learning, which decouples navigation knowledge into shared and specific parts, thereby facilitating new tasks learning using shared knowledge while avoiding forgetting. (b) a Task-Aware Knowledge Aggregation to automatically aggregate the learned knowledge according to a specific navigation task for task specific inference.
• Figure 4: Illustration of the Navigation Chain-of-Thought to design various specific LLM chains of thought for specific instruction style tasks to facilitate the embodied navigation performance.
• Figure 5: Illustration of the lifelong navigation benchmark. We establish a total of 18 navigation tasks for lifelong learning, including 18 navigation scenes and 3 types of user instruction styles (VLN is marked in blue color, OLN is green, and DUN is purple). We use the first 15 tasks for lifelong learning, and the last 3 tasks are used to evaluate unseen scene generalization performance.