Automated Data Curation Using GPS & NLP to Generate Instruction-Action Pairs for Autonomous Vehicle Vision-Language Navigation Datasets

TL;DR

This work addresses the high cost and latency of collecting Instruction-Action pairs for autonomous driving by proposing a GPS-guided, NLP-driven automated pipeline. The authors introduce the ADVLAT-Engine, which synchronizes video, GPS trajectories, and GPS-app outputs (transcribed via OpenAI Whisper) to form vision-language-action triads without human tagging, and they classify GPS instructions into eight referentiality categories. A pilot study collects thousands of commands from Apple Maps, Google Maps, and Waze across diverse routes, demonstrating the feasibility of fully automated IA data generation and annotation. The approach promises scalable, cost-efficient creation of robust vision-language navigation data with broad applicability to VLN models and human-AV interaction scenarios across varied geographic environments.

Abstract

Instruction-Action (IA) data pairs are valuable for training robotic systems, especially autonomous vehicles (AVs), but having humans manually annotate this data is costly and time-inefficient. This paper explores the potential of using mobile application Global Positioning System (GPS) references and Natural Language Processing (NLP) to automatically generate large volumes of IA commands and responses without having a human generate or retroactively tag the data. In our pilot data collection, by driving to various destinations and collecting voice instructions from GPS applications, we demonstrate a means to collect and categorize the diverse sets of instructions, further accompanied by video data to form complete vision-language-action triads. We provide details on our completely automated data collection prototype system, ADVLAT-Engine. We characterize collected GPS voice instructions into eight different classifications, highlighting the breadth of commands and referentialities available for curation from freely available mobile applications. Through research and exploration into the automation of IA data pairs using GPS references, the potential to increase the speed and volume at which high-quality IA datasets are created, while minimizing cost, can pave the way for robust vision-language-action (VLA) models to serve tasks in vision-language navigation (VLN) and human-interactive autonomous systems.

Figures (3)

• Figure 1: The prevalent usage of the Global Positioning System (GPS) navigation service apps forms a plethora of valuable natural language instructions to drivers. In this research, we highlight the unexplored potential of this data for use in the autonomous driving field.
• Figure 2: We demonstrated the capability of forming a fully automatic vision-language-action data generation system, a prototype of our proposed ADVLAT-Engine, using a single mobile phone. By logging GPS positions and recording video, there is sufficient information to synchronize frames to keypoints where GPS instructions are verbalized, extracted by a speech transcription module. Continuous frames between verbal events are available, making it possible for models to learn actions as sequences of video frame states or trajectory waypoints. As illustrated, language from the GPS system is instructive and scene-referential. In this demo, we synchronize video from an iPhone's native camera app, spatial positioning logs from the free myTracks app, and verbalized instructions from Apple Maps converted to text using OpenAI's whisper transcription model.
• Figure 3: System overview of the ADVLAT-Engine. The method used to collect data consisted of storing the text directions provided by the GPS, and recording the verbal instructions spoken aloud. These pairs are then manually annotated and sorted into various categories depending on the types on references made in the instruction, shown in the following tables.