PWC-MoE: Privacy-Aware Wireless Collaborative Mixture of Experts
Yang Su, Na Yan, Yansha Deng, Robert Schober
TL;DR
The paper tackles privacy and bandwidth challenges in deploying large language models by introducing PWC-MoE, a privacy-aware wireless cooperative mixture of experts. It partitions processing between local privacy experts for sensitive tokens and remote non-privacy experts for non-sensitive ones, guided by a sparse gating mechanism and a bandwidth-aware token offloading scheme that uses a learned token-importance predictor. Key contributions include a group-wise load-balancing loss to prevent overloading within expert groups, a Gumbel-Softmax-based routing scheme with privacy isolation, and an importance-driven uplink strategy that preserves performance under limited bandwidth. Experiments on Banking77 with a GPT-2 backbone demonstrate stable convergence, high accuracy with significantly reduced uploaded tokens, and clear advantages of the importance predictor over random selection in bandwidth-constrained settings. The framework offers a practical, scalable solution for privacy-sensitive and resource-constrained LLM deployment in edge-cloud wireless environments.
Abstract
Large language models (LLMs) hosted on cloud servers alleviate the computational and storage burdens on local devices but raise privacy concerns due to sensitive data transmission and require substantial communication bandwidth, which is challenging in constrained environments. In contrast, small language models (SLMs) running locally enhance privacy but suffer from limited performance on complex tasks. To balance computational cost, performance, and privacy protection under bandwidth constraints, we propose a privacy-aware wireless collaborative mixture of experts (PWC-MoE) framework. Specifically, PWC-MoE employs a sparse privacy-aware gating network to dynamically route sensitive tokens to privacy experts located on local clients, while non-sensitive tokens are routed to non-privacy experts located at the remote base station. To achieve computational efficiency, the gating network ensures that each token is dynamically routed to and processed by only one expert. To enhance scalability and prevent overloading of specific experts, we introduce a group-wise load-balancing mechanism for the gating network that evenly distributes sensitive tokens among privacy experts and non-sensitive tokens among non-privacy experts. To adapt to bandwidth constraints while preserving model performance, we propose a bandwidth-adaptive and importance-aware token offloading scheme. This scheme incorporates an importance predictor to evaluate the importance scores of non-sensitive tokens, prioritizing the most important tokens for transmission to the base station based on their predicted importance and the available bandwidth. Experiments demonstrate that the PWC-MoE framework effectively preserves privacy and maintains high performance even in bandwidth-constrained environments, offering a practical solution for deploying LLMs in privacy-sensitive and bandwidth-limited scenarios.