All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptation

Abstract

Deploying vision-and-language navigation (VLN) agents requires adaptation across diverse scenes and environments, but fine-tuning on a specific scenario often causes catastrophic forgetting in others, which severely limits flexible long-term deployment. We formalize this challenge as the all-day multi-scenes lifelong VLN (AML-VLN) problem. Existing parameter-efficient adapters (e.g., LoRA and its variants) are limited by their two-dimensional matrix form, which fails to capture the multi-hierarchical navigation knowledge spanning multiple scenes and environments. To address this, we propose Tucker Adaptation (TuKA), which represents the multi-hierarchical navigation knowledge as a high-order tensor and leverages Tucker decomposition to decouple the knowledge into shared subspaces and scenario-specific experts. We further introduce a decoupled knowledge incremental learning strategy to consolidate shared subspaces while constraining specific experts for decoupled lifelong learning. Building on TuKA, we also develop a VLN agent named AlldayWalker, which continually learns across multiple navigation scenarios, achieving all-day multi-scenes navigation. Extensive experiments show that AlldayWalker consistently outperforms state-of-the-art baselines.

Figures (16)

• Figure 1: Illustration of the proposed all-day multi-scenes lifelong vision-and-language navigation learning and Tucker Adaptation (TuKA). It requires VLN agents to continually learn across multiple scenes and diverse environments (low-light, overexposure, and scattering), progressively consolidating navigation knowledge to achieve all-day multi-scenes navigation. Different from LoRA and its variants, which only perform continual learning with single-dimensional task knowledge, our proposed TuKA decouples and represents multi-hierarchical task knowledge in a high-order tensor.
• Figure 2: Illustration of catastrophic forgetting in lifelong navigation learning. The new scenario adaptation leads to catastrophic forgetting of old scenarios.
• Figure 3: Illustration for comparison of existing LoRAs and our TuKA architecture. Different from the LoRA or MoE-LoRA variants, which represent simple knowledge within a two-hierarchical matrix, TuKA decoupling represents the multi-hierarchical knowledge within a high-order tensor.
• Figure 4: Illustration of the proposed decoupled knowledge incremental learning. Our TuKA performs decoupled incremental learning for multi-hierarchical knowledge in a high-dimensional space.
• Figure 5: Illustration of navigation scenario examples of the Allday-Habitat simulation platform. It includes four common environments: (a) normal, (b) low-light, (c) overexposure, and (d) scattering.