pFedLVM: A Large Vision Model (LVM)-Driven and Latent Feature-Based Personalized Federated Learning Framework in Autonomous Driving

TL;DR

The paper tackles poor generalization in autonomous driving semantic segmentation under domain shift and growing data volumes by integrating a server-side Large Vision Model with latent-feature exchange in a personalized federated learning framework.pFedLVM deploys the LVM backbone exclusively on the central server and uses a feature compressor on vehicles to send compact latent features, significantly reducing communication and computation while enabling personalization through vehicle-specific compressors and SSeg heads.Empirical results on Cityscapes and CamVid demonstrate that pFedLVM surpasses state-of-the-art FL baselines in mIoU, mF1, mPrecision, and mRecall, with notable gains from personalization and efficient latent-feature sharing, as well as improved class separation in t-SNE visualizations.The framework offers a practical approach to scalable, privacy-preserving AD systems with strong generalization in non-IID settings and paves the way for extensions to multi-modal data in intelligent transportation.

Abstract

Deep learning-based Autonomous Driving (AD) models often exhibit poor generalization due to data heterogeneity in an ever domain-shifting environment. While Federated Learning (FL) could improve the generalization of an AD model (known as FedAD system), conventional models often struggle with under-fitting as the amount of accumulated training data progressively increases. To address this issue, instead of conventional small models, employing Large Vision Models (LVMs) in FedAD is a viable option for better learning of representations from a vast volume of data. However, implementing LVMs in FedAD introduces three challenges: (I) the extremely high communication overheads associated with transmitting LVMs between participating vehicles and a central server; (II) lack of computing resource to deploy LVMs on each vehicle; (III) the performance drop due to LVM focusing on shared features but overlooking local vehicle characteristics. To overcome these challenges, we propose pFedLVM, a LVM-Driven, Latent Feature-Based Personalized Federated Learning framework. In this approach, the LVM is deployed only on central server, which effectively alleviates the computational burden on individual vehicles. Furthermore, the exchange between central server and vehicles are the learned features rather than the LVM parameters, which significantly reduces communication overhead. In addition, we utilize both shared features from all participating vehicles and individual characteristics from each vehicle to establish a personalized learning mechanism. This enables each vehicle's model to learn features from others while preserving its personalized characteristics, thereby outperforming globally shared models trained in general FL. Extensive experiments demonstrate that pFedLVM outperforms the existing state-of-the-art approaches.

Figures (15)

• Figure 1: The Federated AD (FedAD) system. As time progresses, the virtual dataset that needs to be fitted by FedAD model expands continuously.
• Figure 2: Illustration of the strength of larger model nakkiran2021deep.
• Figure 3: The illustration of the proposed pFedLVM framework. The proposed pFedLVM is composed of one central server and $|\mathcal{V}|$ vehicles. For each vehicle (e.g., $Vehicle_v$), the feature compressor extracts the compressed features which are transmitted to central server. The central server then uses LVM as a backbone to extract the shared features of all participating vehicles, and returns the extracted shared features to all involved vehicles. Once each vehicle received the shared features from the central server, the downstream SSeg head, taking such shared features as input, is optimized by using the loss between the output of the SSeg head and ground truth via back propagation, while the feature compressor is updated by using the distance between the shared features and the compressed features via back propagation.
• Figure 4: This figure illustrates the proposed latent feature-based pFL. We utilize the compressed features and the shared features to compute the loss for the feature compressor update. Meanwhile, the SSeg head, taking the shared feature as input, is trained based on the output of the SSeg head and the ground truth. As a result, the feature compressor and the SSeg head both learn shared features while maintaining local uniqueness.
• Figure 5: Illustration of the toy example of pFedLVM.