VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigation

TL;DR

Outdoor Vision-and-Language Navigation suffers from limited training diversity. VLN-Video addresses this by pretraining on driving videos augmented with template-infilled instructions and rotation-based action predictions, using three proxy tasks (Masked Language Modeling, Instruction and Trajectory Matching, Next Action Prediction) before fine-tuning on Touchdown. The approach yields state-of-the-art results on Touchdown, with notable improvements in path accuracy and robust generalization to unseen environments. This work demonstrates that driving videos can meaningfully enrich multimodal navigation models and offers a scalable data-generation framework for VLN pretraining.

Abstract

Outdoor Vision-and-Language Navigation (VLN) requires an agent to navigate through realistic 3D outdoor environments based on natural language instructions. The performance of existing VLN methods is limited by insufficient diversity in navigation environments and limited training data. To address these issues, we propose VLN-Video, which utilizes the diverse outdoor environments present in driving videos in multiple cities in the U.S. augmented with automatically generated navigation instructions and actions to improve outdoor VLN performance. VLN-Video combines the best of intuitive classical approaches and modern deep learning techniques, using template infilling to generate grounded navigation instructions, combined with an image rotation similarity-based navigation action predictor to obtain VLN style data from driving videos for pretraining deep learning VLN models. We pre-train the model on the Touchdown dataset and our video-augmented dataset created from driving videos with three proxy tasks: Masked Language Modeling, Instruction and Trajectory Matching, and Next Action Prediction, so as to learn temporally-aware and visually-aligned instruction representations. The learned instruction representation is adapted to the state-of-the-art navigator when fine-tuning on the Touchdown dataset. Empirical results demonstrate that VLN-Video significantly outperforms previous state-of-the-art models by 2.1% in task completion rate, achieving a new state-of-the-art on the Touchdown dataset.

Figures (6)

• Figure 1: Overview of our proposed method VLN-Video: We annotate driving videos with synthetic navigation instructions by extracting instruction templates from the Touchdown dataset and filling them with actions predicted using our image rotation similarity based navigation action predictor and objects detected using a pre-trained object detector. We pre-train VLN models on both the processed video data and Touchdown data to learn better domain knowledge with three proxy tasks. Lastly, we transfer the learned language representation to VLN via fine-tuning.
• Figure 2: Overview of our proposed method to detect the turn between consecutive frames in driving videos, and generate synthetic instructions with template infilling method.
• Figure 3: Qualitative Analysis of our proposed VLN-Video in generating synthetic instructions for videos in BDD100K dataset. The object entities mentioned in the synthetic instructions generated with our VLN-Video are in red, and their corresponding location is bounded in the red box in the frames. The turn actions are in blue.
• Figure 4: Qualitative Analysis of our proposed VLN-Video in learning richer visual objects to help decision making during navigation. Symbols in green are the actions made by our method, and symbols in blue are the actions made by the baseline method.
• Figure 5: Qualitative Analysis of our proposed VLN-Video in learning richer visual objects to help decision making during navigation. Symbols in green are the actions made by our method, and symbols in blue are the actions made by the baseline method.