CYCle: Choosing Your Collaborators Wisely to Enhance Collaborative Fairness in Decentralized Learning

TL;DR

CYCle tackles collaborative fairness in private decentralized learning by introducing a reputation-based mechanism that uses gradient alignment between local cross-entropy loss and pairwise distillation losses to selectively share knowledge. It defines mean collaboration gain (MCG) and collaboration gain spread (CGS) as a fairness objective, and demonstrates that maximizing MCG while minimizing CGS yields fairer outcomes under data heterogeneity. The protocol accommodates private data via CaPriDe-like encryption, and extends to Gossip-SGD with a dynamic, reputation-driven mixing matrix. Theoretical guarantees in a two-client mean-estimation setting show CYCle’s robustness to heterogeneity, while extensive experiments across CIFAR and Fed-ISIC2019 confirm improved fairness with maintained utility, including strong reputation-reward correlations. Overall, CYCle offers a practical, privacy-preserving approach to fair collaboration in decentralized learning, balancing utility and fairness in heterogeneous networks and scalable to gossip-based topologies.

Abstract

Collaborative learning (CL) enables multiple participants to jointly train machine learning (ML) models on decentralized data sources without raw data sharing. While the primary goal of CL is to maximize the expected accuracy gain for each participant, it is also important to ensure that the gains are fairly distributed: no client should be negatively impacted, and gains should reflect contributions. Most existing CL methods require central coordination and focus only on gain maximization, overlooking fairness. In this work, we first show that the existing measure of collaborative fairness based on the correlation between accuracy values without and with collaboration has drawbacks because it does not account for negative collaboration gain. We argue that maximizing mean collaboration gain (MCG) while simultaneously minimizing the collaboration gain spread (CGS) is a fairer alternative. Next, we propose the CYCle protocol that enables individual participants in a private decentralized learning (PDL) framework to achieve this objective through a novel reputation scoring method based on gradient alignment between the local cross-entropy and distillation losses. We further extend the CYCle protocol to operate on top of gossip-based decentralized algorithms such as Gossip-SGD. We also theoretically show that CYCle performs better than standard FedAvg in a two-client mean estimation setting under high heterogeneity. Empirical experiments demonstrate the effectiveness of the CYCle protocol to ensure positive and fair collaboration gain for all participants, even in cases where the data distributions of participants are highly skewed.

Figures (27)

• Figure 1: Illustration of the proposed Choose Your Collaborators Wisely (CYCle) protocol for Private Decentralized Learning (PDL).
• Figure 1: Performance comparison on CIFAR-10 dataset: Validation accuracy evaluated with $N=5$ participants. The top section of the table presents the performance of our proposed framework compared to the FedAvg algorithm. The bottom part of the table compares the collaboration gain and fairness of our proposed CYCle algorithm with existing works (rows 6-9). We use MVA ($\uparrow$), MCG ($\uparrow$) and CGS ($\downarrow$) as eval. metrics.
• Figure 2: Illustration of confidential and private decentralized (CaPriDe) learning framework for $N$ participants, showing the learning process for only participant $\mathcal{P}_1$. Here, $\mathcal{D}_1 = (\mathcal{X}_1, \mathcal{Y}_1) = \{{\bm{x}}_{j,1},y_{j,1}\}_{j=1}^{|{\mathcal{D}}_1|}$ is the local data of ${\mathcal{P}}_1$, $\color{blue} \mathcal{E}(\mathcal{X}_1)$ denotes the collection of encrypted unlabeled samples of $\mathcal{P}_1$, and ${\bm{p}}_{(1,l)}$ and ${\bm{z}}_{(1,l)}$ are the prediction probabilities and logits obtained by applying model $\mathcal{M}_{\theta_l}$ of participant $\mathcal{P}_l$ to ${\mathcal{X}}_1$. Black dashed line represents exchange of encrypted data between participants in each round, and blue dashed line denotes a single transfer at the beginning of the protocol. ${\bm{d}}$ is a decryption method. $\mathcal{L}_{CE}$ and $\mathcal{L}_{DL}$ represent the cross-entropy and distillation losses, respectively. In the context of our paper, $\mathcal{L}_{{DL}_{(1,k)}} \coloneq \widehat{\mathcal{L}}_{{KL}_1} - {\bm{d}} \left(\mathcal{E} \left( \widetilde{\mathcal{L}}_{{KL}_{(1,k)}} \right) \right)$.
• Figure 3: Validation accuracies of participants ($\mathcal{P}_1$, ..., $\mathcal{P}_5$) with $\mathcal{P}_5$ having varying rates of label flipping. The last column refers to the average accuracy of honest participants ($\mathcal{P}_1$, ..., $\mathcal{P}_4$). The corresponding figure with reputation scores: Fig. \ref{['fig: free-rider-study']}.
• Figure 4: Per-participant performance comparison on CIFAR-10 dataset using custom CNN architecture from xu2021gradient: Validation accuracy evaluated with $N=5$ participants.

Theorems & Definitions (6)

• Definition 1: Collaborative Fairness
• Theorem 4.1
• Lemma A.1
• proof
• Theorem A.1: Restated Theorem \ref{['theorem: cycle-main']}
• proof