Secure Aggregation Meets Sparsification in Decentralized Learning

TL;DR

The paper addresses privacy and communication efficiency in decentralized learning by combining secure aggregation with sparsification. It introduces CESAR, a protocol that enables secure aggregation in sparsified DL through pairwise masking over two-hop neighborhoods, with formal privacy guarantees and collusion-resilience via a masking threshold. The authors provide analytical insights into the expected sharing of parameters under collusion, and validate CESAR experimentally on 48-node, k-regular networks across CIFAR-10, CelebA, and MovieLens, showing accuracy comparable to $D$-PSGD and favorable versus TopK with modest data overhead. This work demonstrates that secure aggregation can be effectively integrated with sparsification in a DL setting, offering a practical path toward privacy-preserving, communication-efficient decentralized learning without relying on a central server.

Abstract

Decentralized learning (DL) faces increased vulnerability to privacy breaches due to sophisticated attacks on machine learning (ML) models. Secure aggregation is a computationally efficient cryptographic technique that enables multiple parties to compute an aggregate of their private data while keeping their individual inputs concealed from each other and from any central aggregator. To enhance communication efficiency in DL, sparsification techniques are used, selectively sharing only the most crucial parameters or gradients in a model, thereby maintaining efficiency without notably compromising accuracy. However, applying secure aggregation to sparsified models in DL is challenging due to the transmission of disjoint parameter sets by distinct nodes, which can prevent masks from canceling out effectively. This paper introduces CESAR, a novel secure aggregation protocol for DL designed to be compatible with existing sparsification mechanisms. CESAR provably defends against honest-but-curious adversaries and can be formally adapted to counteract collusion between them. We provide a foundational understanding of the interaction between the sparsification carried out by the nodes and the proportion of the parameters shared under CESAR in both colluding and non-colluding environments, offering analytical insight into the working and applicability of the protocol. Experiments on a network with 48 nodes in a 3-regular topology show that with random subsampling, CESAR is always within 0.5% accuracy of decentralized parallel stochastic gradient descent (D-PSGD), while adding only 11% of data overhead. Moreover, it surpasses the accuracy on TopK by up to 0.3% on independent and identically distributed (IID) data.

Figures (7)

• Figure 1: Overview of CESAR model masking on $N_1$. (1) Nodes $N_1$, $N_2$ and $N_3$ independently sparsify their local models; (2) $N_1$ masks the common indices of its model with those of neighbors that are 2 hops away (, $N_2$ and $N_3$) and that share a common neighbor ($N_r$); (3) Indices that were kept unmasked are discarded; (4) $N_1$ sends the resulting masked model to $N_r$.
• Figure 2: Variation of the fraction of parameters shared in CESAR with the $\operatorname{deg}$ denoting the degree of the receiving node and $\alpha$ being the fraction of model parameters selected in sparsification by each node for masking requirement $s=1$ (, in the absence of any colluding adversarial nodes). Vertical lines and solid curves correspond to scenarios assessed in our empirical evaluation in \ref{['sec:eval']}.
• Figure 3: Comparison of performance and communication cost of CESAR against D-PSGD with matching configuration over multiple datasets. CESAR is overlapping D-PSGD in the performance plots
• Figure 4: The accuracy comparison of CESAR against D-PSGD over various configurations, data distributions and network degrees
• Figure 5: Amount of data transferred per node in CESAR prestep during training for different graph degrees

Theorems & Definitions (13)

• Theorem 1
• proof
• Lemma 1
• proof
• Lemma 1
• proof
• Theorem 2
• proof
• Theorem 3
• proof