Reducing Communication for Split Learning by Randomized Top-k Sparsification

TL;DR

This work tackles the high communication cost of split learning in vertical federated settings by evaluating several compression strategies and introducing randomized top-$k$ sparsification (RandTopk). RandTopk selects top-$k$ elements with high probability while also sampling non-top-$k$ elements with probability $\alpha$, mitigating local minima and balancing neuron usage to improve convergence and generalization under fixed compression. The authors show, across multiple datasets and models, that RandTopk consistently outperforms size reduction, standard top-$k$, quantization, and L1 regularization at the same compression level, approaching or even matching non-sparse performance in some cases. The results demonstrate RandTopk as a practical method to reduce inter-party communication in split learning while maintaining strong predictive accuracy, with privacy considerations discussed and future work proposed to combine quantization and sparsification further.

Abstract

Split learning is a simple solution for Vertical Federated Learning (VFL), which has drawn substantial attention in both research and application due to its simplicity and efficiency. However, communication efficiency is still a crucial issue for split learning. In this paper, we investigate multiple communication reduction methods for split learning, including cut layer size reduction, top-k sparsification, quantization, and L1 regularization. Through analysis of the cut layer size reduction and top-k sparsification, we further propose randomized top-k sparsification, to make the model generalize and converge better. This is done by selecting top-k elements with a large probability while also having a small probability to select non-top-k elements. Empirical results show that compared with other communication-reduction methods, our proposed randomized top-k sparsification achieves a better model performance under the same compression level.

Figures (8)

• Figure 1: Overview of split learning.
• Figure 2: The loss surface, gradient field, and learning trajectory of the toy example.
• Figure 3: Convergence speed of different methods under certain compression levels. Compressed rates can be found in \ref{['table:main-result']}. Top row: accuracy vs. #epochs. Bottom row: Accuracy vs. communication (total size of the message transferred, vanilla split learning in one epoch = 1). Methods that are not applicable/fail to converge under the given compression level are omitted.
• Figure 4: Training loss and generalization error on CIFAR-100. The compressed size is 2.86%.
• Figure 5: Distribution of top-$k$ neurons during inference.