CLIP: Client-Side Invariant Pruning for Mitigating Stragglers in Secure Federated Learning

TL;DR

This paper tackles the straggler problem in secure federated learning by introducing CLIP, a client-side invariant pruning framework that preserves privacy under SecAgg while accelerating training. CLIP comprises straggler self-identification, invariant neuron pruning based on relative weight changes with a principled sub-model size $p_k = \frac{t_k}{T}$, and network-aware pruning to mitigate network delays without altering update sizes. By performing pruning entirely on the client side, CLIP remains compatible with secure aggregation and padding ensures identical update dimensions, preserving privacy assumptions. Empirical results on CIFAR10, Shakespeare, and FEMNIST show speedups from $13\%$ to $34\%$ with accuracy losses capped at $2.6\%$, indicating practical impact for privacy-preserving FL in heterogeneous environments.

Abstract

Secure federated learning (FL) preserves data privacy during distributed model training. However, deploying such frameworks across heterogeneous devices results in performance bottlenecks, due to straggler clients with limited computational or network capabilities, slowing training for all participating clients. This paper introduces the first straggler mitigation technique for secure aggregation with deep neural networks. We propose CLIP, a client-side invariant neuron pruning technique coupled with network-aware pruning, that addresses compute and network bottlenecks due to stragglers during training with minimal accuracy loss. Our technique accelerates secure FL training by 13% to 34% across multiple datasets (CIFAR10, Shakespeare, FEMNIST) with an accuracy impact of between 1.3% improvement to 2.6% reduction.

Figures (7)

• Figure 1: (a) Secure aggregation in federated learning preserves privacy of client training data; however stragglers with slow compute or network bandwidth increase training time. (b) State-of-the-art server-side solutions for straggler mitigation are incompatible with secure aggregation and do not provide privacy. (c) Our solution enables client-side invariant neuron pruning and network-aware pruning that improves training speed while maintaining accuracy and privacy.
• Figure 2: Time per training round for FL with secure aggregation in the presence of stragglers (clients with slower compute and lower network bandwidths). With compute stragglers (with 33% slower compute), training round time increases by 24.2%, but ideally, pruning straggler models has the potential to reduce this to within 1%. For network stragglers, with less bandwidth (4G), the training time increases by 12.7%, but dropout cannot naively address this overhead due to SecAgg constraints.
• Figure 3: Overview of CLIP. It consists of (a) a straggler-self identification protocol on the client-side, (b) client-side invariant neuron selection algorithm for minimal accuracy impact and maintaining privacy (compatibility with SecAgg), and (c) compute and network aware pruning for performance.
• Figure 4: Accuracy versus sub-model size of CLIP compared to Ordered and Random Dropout and SecAgg (baseline). Averaged over 3 runs, where each run is trained for 100 rounds with 20 clients.
• Figure 5: Accuracy versus training time for CLIP compared to Ordered dropout, Random dropout and SecAgg (baseline) after 100 rounds, with 20 clients of which 4 are stragglers.