Federated Learning Meets LLMs: Feature Extraction From Heterogeneous Clients
Abdelrhman Gaber, Hassan Abd-Eltawab, Youssif Abuzied, Muhammad ElMahdy, Tamer ElBatt
TL;DR
This work tackles data view heterogeneity in federated learning by using pretrained large language models (LLMs) as universal feature extractors. Tabular records are serialized to text and passed through frozen LLM backbones to generate embeddings that standardize heterogeneous feature spaces, enabling lightweight on-device classifiers trained via FedAvg. Empirical results on Framingham CHD data show that FedLLM-Align delivers higher F1-scores than strong baselines, while substantially reducing communication cost and maintaining stable convergence under varied schema overlaps. The approach offers a privacy-preserving, scalable solution for cross-institution FL in heterogeneous environments, with ClinicalBERT achieving top-domain accuracy and DistilBERT offering a favorable accuracy-efficiency balance. Future work includes exploring larger federated networks, adapter-based fine-tuning, and domain expansion to finance and IoT.
Abstract
Federated learning (FL) enables collaborative model training without sharing raw data, making it attractive for privacy-sensitive domains such as healthcare, finance, and IoT. A major obstacle, however, is the heterogeneity of tabular data across clients, where divergent schemas and incompatible feature spaces prevent straightforward aggregation. To address this challenge, we propose FedLLM-Align, a federated framework that leverages pre-trained large language models (LLMs) as universal feature extractors. Tabular records are serialized into text, and embeddings from models such as DistilBERT, ALBERT, RoBERTa, and ClinicalBERT provide semantically aligned representations that support lightweight local classifiers under the standard FedAvg protocol. This approach removes the need for manual schema harmonization while preserving privacy, since raw data remain strictly local. We evaluate FedLLM-Align on coronary heart disease prediction using partitioned Framingham datasets with simulated schema divergence. Across all client settings and LLM backbones, our method consistently outperforms state-of-the-art baselines, achieving up to +0.25 improvement in F1-score and a 65% reduction in communication cost. Stress testing under extreme schema divergence further demonstrates graceful degradation, unlike traditional methods that collapse entirely. These results establish FedLLM-Align as a robust, privacy-preserving, and communication-efficient solution for federated learning in heterogeneous environments.