Learning to Drive Anywhere with Model-Based Reannotation

TL;DR

The paper presents Model-Based ReAnnotation (MBRA), a two-stage framework that leverages abundant passive data to train robust long-horizon visual navigation policies. A short-horizon relabeler generates high-quality actions from noisy trajectories via a differentiable forward model, which are then distilled into LogoNav, a long-horizon, goal-pose conditioned policy trained by imitation on the relabeled data. The approach enables scaling from crowd-sourced data and action-free videos to 300+ meter navigation in unseen indoor and outdoor environments, demonstrated across robots and six cities on three continents. MBRA outperforms baselines such as NoMaD and GCP, and proves effective with action-free YouTube data, addressing the data bottleneck in robotic navigation. The work highlights practical impact for affordable, broadly deployable navigation policies and provides a foundation for future enhancements in 3D geometry and human-aware navigation in crowds.

Abstract

Developing broadly generalizable visual navigation policies for robots is a significant challenge, primarily constrained by the availability of large-scale, diverse training data. While curated datasets collected by researchers offer high quality, their limited size restricts policy generalization. To overcome this, we explore leveraging abundant, passively collected data sources, including large volumes of crowd-sourced teleoperation data and unlabeled YouTube videos, despite their potential for lower quality or missing action labels. We propose Model-Based ReAnnotation (MBRA), a framework that utilizes a learned short-horizon, model-based expert model to relabel or generate high-quality actions for these passive datasets. This relabeled data is then distilled into LogoNav, a long-horizon navigation policy conditioned on visual goals or GPS waypoints. We demonstrate that LogoNav, trained using MBRA-processed data, achieves state-of-the-art performance, enabling robust navigation over distances exceeding 300 meters in previously unseen indoor and outdoor environments. Our extensive real-world evaluations, conducted across a fleet of robots (including quadrupeds) in six cities on three continents, validate the policy's ability to generalize and navigate effectively even amidst pedestrians in crowded settings.

Figures (8)

• Figure -1: We train a highly generalizable navigation policy that can control robots in a variety of conditions and be deployed zero-shot in new environments across the world. Our proposed method, Model-Based ReAnnotation, enables imitation learning from noisy, passive data, such as low-quality crowd-sourced demonstrations or even videos from the web.
• Figure 0: Overview of MBRA. We propose a two-step process: In the first stage, we train a short-horizon reannotation policy with a robust MBL approach on the noisy dataset, which can be used for short-horizon image-conditioned navigation and which we leverage to relabel the noisy dataset with improved action labels. In step 2, we train a long-horizon navigation policy with the generated action labels.
• Figure 1: The aim of MBRA is to relabel low-quality data with actions that are better than the actions in the dataset, in the sense that they more effectively link states over short-horizon trajectory snippets. Compared to methods that use one- or multi- step inverse models (e.g., VPT, GCP) or the original noisy actions, training on actions from MBRA leads to significantly more effective policies.
• Figure 2: Network architecture. In addition to the visual observations, We feed the delay step and the previous actions to consider the system delay in the MBL objective. For the long-horizon navigation policy, we replace the visual encoder for the current and the goal observation with the MLP layers for the goal pose.
• Figure 3: Overview of the robot hardwares and systems. ERZ can be controlled over a internet connection for data collection and for deploying our navigation policy. Vizbot and Go1 with different cameras are controlled from the onboard robot controller with ROS.