MAPL: Model Agnostic Peer-to-peer Learning

TL;DR

MAPL addresses decentralized learning with heterogeneous client models by jointly learning personalized models and a privacy-preserving collaboration graph in a peer-to-peer setting. It introduces PML, which uses inter- and intra-client contrastive losses plus learnable class prototypes to align representations, and CGL, which derives edge weights from classifier-head similarities under a sparsity-promoting graph regularizer. The method alternates local PML updates with graph-learning steps, enabling dynamic neighborhood discovery without a central server; experiments on CIFAR-10, CINIC-10, SVHN, MNIST, and FashionMNIST show MAPL is competitive with or superior to centralized model-agnostic baselines and that the learned graphs reveal meaningful data-distribution clusters. The approach reduces communication overhead through sparsified graphs and provides a practical, privacy-conscious path toward scalable decentralized learning in heterogeneous environments.

Abstract

Effective collaboration among heterogeneous clients in a decentralized setting is a rather unexplored avenue in the literature. To structurally address this, we introduce Model Agnostic Peer-to-peer Learning (coined as MAPL) a novel approach to simultaneously learn heterogeneous personalized models as well as a collaboration graph through peer-to-peer communication among neighboring clients. MAPL is comprised of two main modules: (i) local-level Personalized Model Learning (PML), leveraging a combination of intra- and inter-client contrastive losses; (ii) network-wide decentralized Collaborative Graph Learning (CGL) dynamically refining collaboration weights in a privacy-preserving manner based on local task similarities. Our extensive experimentation demonstrates the efficacy of MAPL and its competitive (or, in most cases, superior) performance compared to its centralized model-agnostic counterparts, without relying on any central server. Our code is available and can be accessed here: https://github.com/SayakMukherjee/MAPL

Figures (9)

• Figure 1: Illustration of the local training methodology of MAPL.
• Figure 2: (Left) Computing similarity between clients. (Right) Evolution of collaboration graph: circles for clients, colors for classes.
• Figure 3: (Left) Data distribution and similarity of classifier weights trained without (Middle) and with (Right)$l_\text{proto}$ for aligning the class-wise feature representations.
• Figure 4: Client data distribution, learned collaboration graph (part (b)) and latent embedding (part (c)) with $M=10$ for Scenario $1$.
• Figure 5: Client data distribution, learned collaboration graph (part (b)) and latent embedding (part (c)) with $M=10$ for Scenario $1$.