UrbanNav: Learning Language-Guided Urban Navigation from Web-Scale Human Trajectories

TL;DR

UrbanNav tackles the challenge of following free-form natural language instructions in real-world cities by leveraging web-scale walking videos to train language-guided navigation policies. It introduces an automated data pipeline that extracts robot-compatible egocentric trajectories, annotates landmarks, and generates descriptive instructions, yielding over 1,500 hours of data and 3 million instruction-trajectory-landmark triplets. The model fuses language and vision through FiLM-conditioned features and a Transformer to predict multi-step waypoints, trained with a composite loss and evaluated offline and on real robots. Results show state-of-the-art performance, strong generalization to unseen cities, and robustness to noisy instructions, underscoring the viability of web-scale data for real-world embodied navigation.

Abstract

Navigating complex urban environments using natural language instructions poses significant challenges for embodied agents, including noisy language instructions, ambiguous spatial references, diverse landmarks, and dynamic street scenes. Current visual navigation methods are typically limited to simulated or off-street environments, and often rely on precise goal formats, such as specific coordinates or images. This limits their effectiveness for autonomous agents like last-mile delivery robots navigating unfamiliar cities. To address these limitations, we introduce UrbanNav, a scalable framework that trains embodied agents to follow free-form language instructions in diverse urban settings. Leveraging web-scale city walking videos, we develop an scalable annotation pipeline that aligns human navigation trajectories with language instructions grounded in real-world landmarks. UrbanNav encompasses over 1,500 hours of navigation data and 3 million instruction-trajectory-landmark triplets, capturing a wide range of urban scenarios. Our model learns robust navigation policies to tackle complex urban scenarios, demonstrating superior spatial reasoning, robustness to noisy instructions, and generalization to unseen urban settings. Experimental results show that UrbanNav significantly outperforms existing methods, highlighting the potential of large-scale web video data to enable language-guided, real-world urban navigation for embodied agents.

Figures (5)

• Figure 1: Overview of Our UrbanNav framework. UrbanNav is designed to tackle the challenging task of language-guided urban navigation. Its scalable data pipeline constructs a large dataset from web-scale human walking videos. Our policy is trained on this dataset and fine-tuned with a small amount of real-world data, enabling it to interpret complex natural language instructions and navigate challenging, unseen urban environments.
• Figure 2: UrbanNav Data Construction Pipeline. The process is divided into three main steps: 1) Trajectory Annotation, where human walking videos are preprocessed, segmented, and then annotated with a camera pose estimator to generate trajectories. 2) Robot-Compatible Data Filtering, where low-quality segments with pitch, yaw, or dense crowd issues are automatically filtered out. 3) Language Instruction Annotation, where a large language model is used to generate rich, descriptive language instructions for each trajectory, along with landmark bounding boxes and depth maps.
• Figure 3: Overall Illustration of UrbanNav. The model takes historical images and a language instruction as input, fuses their features, and uses a Transformer to predict future frame features, waypoints, directions, and an arrival status.
• Figure 4: Qualitative Results. The figures show trajectory visualizations in four different scenarios. For each set of images, the left side represents the initial observation and instruction, while the right side shows the real-robot trajectories and target positions in the world coordinate system for different methods.
• Figure 5: Impact of Web data scaling. The figures show the performance of UrbanNav on unseen environments as a function of the training data size.