Edge-Cloud Collaborative Motion Planning for Autonomous Driving with Large Language Models

TL;DR

The paper tackles real-time motion planning for autonomous driving using large language models, addressing edge latency and cloud compute constraints under dynamic open-world conditions. It proposes EC-Drive, a two-tier system that leverages edge LLaMA-Adapter-based reasoning for routine tasks and cloud-scale GPT-4 inference for complex scenarios, with data-drift detection via Alibi Detect to selectively offload samples. The work contributes a novel edge-cloud framework, multimodal integration of linguistic and visual cues, and comprehensive experimental validation showing reduced latency and improved handling of new obstacles. The results demonstrate practical feasibility for LVLM-enabled autonomous driving and provide guidance for deploying such systems in real-world environments, including dataset-specific insights from Guangzhou.

Abstract

Integrating large language models (LLMs) into autonomous driving enhances personalization and adaptability in open-world scenarios. However, traditional edge computing models still face significant challenges in processing complex driving data, particularly regarding real-time performance and system efficiency. To address these challenges, this study introduces EC-Drive, a novel edge-cloud collaborative autonomous driving system with data drift detection capabilities. EC-Drive utilizes drift detection algorithms to selectively upload critical data, including new obstacles and traffic pattern changes, to the cloud for processing by GPT-4, while routine data is efficiently managed by smaller LLMs on edge devices. This approach not only reduces inference latency but also improves system efficiency by optimizing communication resource use. Experimental validation confirms the system's robust processing capabilities and practical applicability in real-world driving conditions, demonstrating the effectiveness of this edge-cloud collaboration framework. Our data and system demonstration will be released at https://sites.google.com/view/ec-drive.

Figures (6)

• Figure 1: Architecture of the EC-Drive system. LLM-based motion planning is performed on edge devices within the vehicle, while complex inference tasks are offloaded to the cloud, which has larger models and more extensive resources.
• Figure 2: Motion planning process on the edge through large language models, utilizing vision and LiDAR data for real-time decision-making and execution. ROS stands for Robot Operating System, which is used to execute actions and provide feedback on the execution results.
• Figure 3: Instruction tuning of pretrained LLaMA2 models for autonomous driving, using multi-view images and prompt for efficient adaptation to specific driving scenarios.
• Figure 4: Edge model performs step-by-step reasoning and decision making in a complex traffic environment
• Figure 5: The cloud model addresses incremental driving scenarios, and the yellow dotted line shows the logical dependencies between stages.