Towards Efficient Communication and Secure Federated Recommendation System via Low-rank Training

TL;DR

The paper tackles the bottleneck of communication in Federated Recommendation by introducing Correlated Low-rank Structure (CoLR), a framework that updates only a small, trainable low-rank component while freezing the rest of the model. CoLR enables additive, HE-friendly aggregation and reduces uplink/downlink traffic by representing local updates as products of small matrices, Δ_Q^{(t)} = B^{(t)} ∑_u A_u^{(t)}, with a shared B across clients. A variant, SCoLR, accommodates heterogeneous network conditions by allowing per-client local ranks and subsampling, preserving performance under varying uplink budgets. Empirical results on MovieLens-1M and Pinterest show substantial payload reductions (up to 93.75%) with only about an 8% drop in HR/NDCG, and demonstrate compatibility with HE-based secure aggregation, highlighting practical impact for privacy-preserving, communication-efficient FedRec systems.

Abstract

Federated Recommendation (FedRec) systems have emerged as a solution to safeguard users' data in response to growing regulatory concerns. However, one of the major challenges in these systems lies in the communication costs that arise from the need to transmit neural network models between user devices and a central server. Prior approaches to these challenges often lead to issues such as computational overheads, model specificity constraints, and compatibility issues with secure aggregation protocols. In response, we propose a novel framework, called Correlated Low-rank Structure (CoLR), which leverages the concept of adjusting lightweight trainable parameters while keeping most parameters frozen. Our approach substantially reduces communication overheads without introducing additional computational burdens. Critically, our framework remains fully compatible with secure aggregation protocols, including the robust use of Homomorphic Encryption. The approach resulted in a reduction of up to 93.75% in payload size, with only an approximate 8% decrease in recommendation performance across datasets. Code for reproducing our experiments can be found at https://github.com/NNHieu/CoLR-FedRec.

Figures (6)

• Figure 1: Illustration of CoLR at training round $t$. At first, the server conducts aggregation over the local model $A_u^{(t-1, \tau_u)}$ to obtain the global model update $A^{(t)}$. Subsequently, $A^{(t)}$ are transmitted to the clients. The client will update their $Q_u^{(t)}$ using this $A^{(t)}$, then initilizes a new matrix $A_u^{(t,0)}$ and download the matrix $B^{(t)}$ which is sampled at the server and shared between clients. Finally, the client carries out local training and then sends the local model update $A_u^{(t, \tau_u)}$ to the server for the next training round.
• Figure 2: PCA components progression. The figures show the number of components that account for 99% (N99-PCA in green) and 95% (N95-PCA in blue) explained variance of all transfer item embedding matrix across communication rounds on the MovieLens-1M (left) and Pinterest (right) datasets.
• Figure 3: Performance on the MovieLens-1M dataset (Top) and the Pinterest dataset (Bottom). We plot the utilities (HR and NDCG) versus the payload reduction and compare CoLR with the base model with the same transfer size. Each point represents the average recommendation performance on the test set across five random seeds. The shaded areas denote the standard deviation over the mean. The dashed black line presents the largest base model's performance.
• Figure 4: HR and NDCG on MovieLens-1M dataset (Top) and Pinterest Dataset (Bottom). We plot the utilities versus the payload reduction and compare CoLR with other methods with the same payload reduction. The dashed black line presents the base model's performance.
• Figure 5: Performance of SCOLR on MovieLens-1M dataset. We plot the utilities versus the payload size. The dashed black line is the base model's performance.

Theorems & Definitions (3)

• proposition 1: Upper bound the error
• lemma 1: Gaussian Initialization
• theorem 1