EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval

TL;DR

EfficientNav tackles on-device object-goal navigation by addressing two key bottlenecks: limited device memory for KV caches and the reduced planning capability of small LLMs. It introduces discrete memory caching, attention-based memory clustering, and semantics-aware memory retrieval to enable effective zero-shot planning with lightweight models, significantly reducing latency and improving success rates on HM3D. Across extensive ablations and comparisons, EfficientNav demonstrates substantial SR/SPL gains over cloud-based baselines and other zero-shot methods, while achieving real-time latency reductions by orders of magnitude. The approach offers a practical path to private, on-device embodied AI for indoor navigation, with clear avenues for optimization and broader impact considerations.

Abstract

Object-goal navigation (ObjNav) tasks an agent with navigating to the location of a specific object in an unseen environment. Embodied agents equipped with large language models (LLMs) and online constructed navigation maps can perform ObjNav in a zero-shot manner. However, existing agents heavily rely on giant LLMs on the cloud, e.g., GPT-4, while directly switching to small LLMs, e.g., LLaMA3.2-11b, suffer from significant success rate drops due to limited model capacity for understanding complex navigation maps, which prevents deploying ObjNav on local devices. At the same time, the long prompt introduced by the navigation map description will cause high planning latency on local devices. In this paper, we propose EfficientNav to enable on-device efficient LLM-based zero-shot ObjNav. To help the smaller LLMs better understand the environment, we propose semantics-aware memory retrieval to prune redundant information in navigation maps. To reduce planning latency, we propose discrete memory caching and attention-based memory clustering to efficiently save and re-use the KV cache. Extensive experimental results demonstrate that EfficientNav achieves 11.1% improvement in success rate on HM3D benchmark over GPT-4-based baselines, and demonstrates 6.7x real-time latency reduction and 4.7x end-to-end latency reduction over GPT-4 planner. Our code is available on https://github.com/PKU-SEC-Lab/EfficientNav.

Figures (11)

• Figure 1: (a) Local devices (e.g., NVIDIA Jetson AGX Orin) exhibit constrained memory capacity (32GB). (b) Only smaller LLMs, such as Llama-3.2-11b, can be used as the planner due to memory limitations. (c) The KV cache of map information accumulates with navigation steps, exceeding the memory capacity. (d) Recomputing the KV cache incurs long real-time latency.
• Figure 2: Software flow of LLM-based ObjNav.
• Figure 3: (a) Average length of different parts of prompts across navigation steps. Prompt lengths vary slightly across models due to divergent sub-goal selections influencing map complexity. (b) On-device LLM planning latency across navigation steps.
• Figure 4: (a) Attention score on the most promising sub-goal (ground-truth object) in different navigation steps. Attention distribution after (b) 10 steps and (c) 30 steps using LLaVA-34b in sub-goal planning. As the amount of map information increases with the progression of navigation, the LLM can not fully understand the navigation map to focus on the most promising sub-goal. We observe similar problems in other small open-source LLMs such as LLaMA and Vicuna. The ground-truth object is chosen by GPT-4o.
• Figure 5: Overview of EfficientNav. Sim. stands for similarity.