FLAD: Federated Learning for LLM-based Autonomous Driving in Vehicle-Edge-Cloud Networks

TL;DR

FLAD presents a cloud-edge-vehicle Federated Learning framework for LLM-based autonomous driving that addresses mobility, heterogeneity, and memory constraints by integrating Federated Hybrid Data Parallelism (FHDP), a mobility-aware SWIFT scheduler, and cloud-to-edge knowledge distillation via CELLAdapt. The approach partitions training and inference workloads across vehicle, edge, and cloud layers, enabling privacy-preserving, personalized AD models with scalable pipeline parallelism and fault-tolerant mechanisms. Empirical results on Jetson-based testbeds and CARLA-driven simulations show FLAD achieving substantial throughput close to centralized training (about 70%) while delivering region-specific improvements in perception and driving metrics through LLM distillation and personalization. The work demonstrates practical viability for distributed AD model training and knowledge sharing in dynamic vehicular networks, with implications for future extension to drones and robots in embodied AI systems.

Abstract

Large Language Models (LLMs) have impressive data fusion and reasoning capabilities for autonomous driving (AD). However, training LLMs for AD faces significant challenges including high computation transmission costs, and privacy concerns associated with sensitive driving data. Federated Learning (FL) is promising for enabling autonomous vehicles (AVs) to collaboratively train models without sharing raw data. We present Federated LLM-based Autonomous Driving (FLAD), an FL framework that leverages distributed multimodal sensory data across AVs in heterogeneous environment. FLAD has three key innovations: (1) a cloud-edge-vehicle collaborative architecture that reduces communication delay and preserving data privacy; (2) an intelligent parallelized collaborative training with a communication scheduling mechanism that optimizes training efficiency, leveraging end-devices otherwise having insufficient resources for model training; and (3) a knowledge distillation method that personalizes LLM according to heterogeneous edge data. In addition, we prototype FLAD in a testbed with NVIDIA Jetsons, overcoming practical implementation challenges including CPU/GPU memory sharing in resource-constrained devices, dynamic model partitions, and fault-tolerant training.Extensive experimental evaluation demonstrates that FLAD achieves superior end-to-end AD performance while efficiently utilizing distributed vehicular resources, opening up new possibilities for future collaborative AD model training and knowledge sharing.

Figures (10)

• Figure 1: Training procedure in FLAD.
• Figure 2: Inference procedure in FLAD.
• Figure 3: The overview of SWIFT in FHDP.
• Figure 4: FHDP testbed.
• Figure 5: Averaging (a) optimization time and (b) recovery time of SWIFT.