City Navigation in the Wild: Exploring Emergent Navigation from Web-Scale Knowledge in MLLMs

TL;DR

CityNav introduces Sparsely Grounded Long-Range Navigation to probe MLLMs' knowledge-based sequential decision-making in real cities using sparse visual inputs. The core approach, Verbalization of Path (VoP), grounds internal world knowledge by prompting explicit destination, localization, and walking directions, complemented by a compact memory architecture (Markovian Memory, Decision History, Previous Visit). Across four cities and multiple MLLMs, VoP yields consistent improvements over standard reasoning baselines, though performance still varies by city and model, revealing gaps in long-horizon embodied reasoning. The work demonstrates that latent world knowledge in large models can be harnessed for realistic, long-range outdoor navigation, while also pointing to remaining challenges in dynamic grounding and context handling in diverse urban environments.

Abstract

Leveraging multimodal large language models (MLLMs) to develop embodied agents offers significant promise for addressing complex real-world tasks. However, current evaluation benchmarks remain predominantly language-centric or heavily reliant on simulated environments, rarely probing the nuanced, knowledge-intensive reasoning essential for practical, real-world scenarios. To bridge this critical gap, we introduce the task of Sparsely Grounded Visual Navigation, explicitly designed to evaluate the sequential decision-making abilities of MLLMs in challenging, knowledge-intensive real-world environments. We operationalize this task with CityNav, a comprehensive benchmark encompassing four diverse global cities, specifically constructed to assess raw MLLM-driven agents in city navigation. Agents are required to rely solely on visual inputs and internal multimodal reasoning to sequentially navigate 50+ decision points without additional environmental annotations or specialized architectural modifications. Crucially, agents must autonomously achieve localization through interpreting city-specific cues and recognizing landmarks, perform spatial reasoning, and strategically plan and execute routes to their destinations. Through extensive evaluations, we demonstrate that current state-of-the-art MLLMs and standard reasoning techniques (e.g., Chain-of-Thought, Reflection) significantly underperform in this challenging setting. To address this, we propose Verbalization of Path (VoP), which explicitly grounds the agent's internal reasoning by probing an explicit cognitive map (key landmarks and directions toward the destination) from the MLLMs, substantially enhancing navigation success. Project Webpage: https://dwipddalal.github.io/AgentNav/

Figures (4)

• Figure 1: The figure illustrates our proposed Verbalization of Path (VoP) method, which extracts city-scale cognitive maps from MLLMs for city navigation in the wild. The red bounding boxes on the New York map highlight the streets and locations explicitly referenced by the MLLM during its verbal reasoning. Note that, in New York City, street names typically correspond to entire street segments.
• Figure 2: Dataset paths visualization of Vienna. Here the black dot's mark the starting point, and the red blobs mark the destination point.
• Figure 3: Illustration of state transition from $S$ (marked by the yellow dot) to $S+1$ using the agent’s internal reasoning through Verbalization of Path (VoP). At each state, the agent perceives visual cues and references its memory to update decisions and navigation strategy. The purple marker denotes the destination (One World Trade Center), while the green marker indicates the starting point.
• Figure 4: Failure case, the red box highlights the path in which AgentNav gets suck in a loop.