Efficient Federated Learning Tiny Language Models for Mobile Network Feature Prediction

TL;DR

The paper addresses high communication overhead in federated learning for mobile-network QoS prediction. It proposes a method that combines transformer-based tiny language models with NNCodec-based compression within an FL framework to exchange compressed differential updates $ΔW_i$; this enables scalable, privacy-preserving collaboration among network cells. Using the Berlin V2X dataset, it details data preprocessing, a custom tokenizer, and three TLM sizes (409k, 10M, 32M) with corresponding memory footprints, aimed at predicting features like ping and SNR. Experiments over 25 FL rounds show uncompressed baselines reach top-1 accuracy > 89% and perplexity < 1.35, while NNCodec-based compression achieves transparency with under 1% of the original data and negligible degradation, highlighting practical viability for autonomous, mobile-network optimization.

Abstract

In telecommunications, Autonomous Networks (ANs) automatically adjust configurations based on specific requirements (e.g., bandwidth) and available resources. These networks rely on continuous monitoring and intelligent mechanisms for self-optimization, self-repair, and self-protection, nowadays enhanced by Neural Networks (NNs) to enable predictive modeling and pattern recognition. Here, Federated Learning (FL) allows multiple AN cells - each equipped with NNs - to collaboratively train models while preserving data privacy. However, FL requires frequent transmission of large neural data and thus an efficient, standardized compression strategy for reliable communication. To address this, we investigate NNCodec, a Fraunhofer implementation of the ISO/IEC Neural Network Coding (NNC) standard, within a novel FL framework that integrates tiny language models (TLMs) for various mobile network feature prediction (e.g., ping, SNR or band frequency). Our experimental results on the Berlin V2X dataset demonstrate that NNCodec achieves transparent compression (i.e., negligible performance loss) while reducing communication overhead to below 1%, showing the effectiveness of combining NNC with FL in collaboratively learned autonomous mobile networks.

Figures (3)

• Figure 1: Federated averaging differential weight updates $\Delta W_i$.
• Figure 2: Concept scheme of TLM-based cellular feature predictions.
• Figure 3: Coding performance of various $qp$ and sparsity values with a fixed TLM_size of 1, learning rate 3e-4, y-axis limited to values $\geq$82%.