Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Topology-Aware Knowledge Propagation in Decentralized Learning

Mansi Sakarvadia, Nathaniel Hudson, Tian Li, Ian Foster, Kyle Chard

TL;DR

The paper tackles the challenge of propagating out-of-distribution (OOD) knowledge in fully decentralized learning where devices communicate only with neighbors. It introduces topology-aware aggregation using degree and betweenness centrality to weight neighbor models, enabling faster and more reliable OOD knowledge dissemination without sacrificing IID performance. Empirical results across 36 topologies and multiple datasets show that topology-aware methods substantially improve OOD propagation while maintaining IID accuracy, highlighting the critical role of network structure in non-IID distributed learning. This work offers a practical mechanism to enhance OOD generalization in decentralized systems, with implications for robust inference in ad hoc and heterogeneous networks.

Abstract

Decentralized learning enables collaborative training of models across naturally distributed data without centralized coordination or maintenance of a global model. Instead, devices are organized in arbitrary communication topologies, in which they can only communicate with neighboring devices. Each device maintains its own local model by training on its local data and integrating new knowledge via model aggregation with neighbors. Therefore, knowledge is propagated across the topology via successive aggregation rounds. We study, in particular, the propagation of out-of-distribution (OOD) knowledge. We find that popular decentralized learning algorithms struggle to propagate OOD knowledge effectively to all devices. Further, we find that both the location of OOD data within a topology, and the topology itself, significantly impact OOD knowledge propagation. We then propose topology-aware aggregation strategies to accelerate (OOD) knowledge propagation across devices. These strategies improve OOD data accuracy, compared to topology-unaware baselines, by 123% on average across models in a topology.

Topology-Aware Knowledge Propagation in Decentralized Learning

TL;DR

The paper tackles the challenge of propagating out-of-distribution (OOD) knowledge in fully decentralized learning where devices communicate only with neighbors. It introduces topology-aware aggregation using degree and betweenness centrality to weight neighbor models, enabling faster and more reliable OOD knowledge dissemination without sacrificing IID performance. Empirical results across 36 topologies and multiple datasets show that topology-aware methods substantially improve OOD propagation while maintaining IID accuracy, highlighting the critical role of network structure in non-IID distributed learning. This work offers a practical mechanism to enhance OOD generalization in decentralized systems, with implications for robust inference in ad hoc and heterogeneous networks.

Abstract

Decentralized learning enables collaborative training of models across naturally distributed data without centralized coordination or maintenance of a global model. Instead, devices are organized in arbitrary communication topologies, in which they can only communicate with neighboring devices. Each device maintains its own local model by training on its local data and integrating new knowledge via model aggregation with neighbors. Therefore, knowledge is propagated across the topology via successive aggregation rounds. We study, in particular, the propagation of out-of-distribution (OOD) knowledge. We find that popular decentralized learning algorithms struggle to propagate OOD knowledge effectively to all devices. Further, we find that both the location of OOD data within a topology, and the topology itself, significantly impact OOD knowledge propagation. We then propose topology-aware aggregation strategies to accelerate (OOD) knowledge propagation across devices. These strategies improve OOD data accuracy, compared to topology-unaware baselines, by 123% on average across models in a topology.
Paper Structure (24 sections, 4 equations, 21 figures, 5 tables, 1 algorithm)

This paper contains 24 sections, 4 equations, 21 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Decentralized Learning Overview
  3. The Problem: Propagating IID vs. OOD Knowledge
  4. Proposed Solution: Topology-Aware Aggregation
  5. Experiments
  6. Topology-Aware vs. Topology-Unaware Aggregation
  7. Impact of Data location
  8. Impact of Topology
  9. Related Work
  10. Conclusion & Future Work
  11. Limitations
  12. Appendix / Supplemental Material
  13. Experiment Setup
  14. Topologies
  15. Training Data Distributions
  16. ...and 9 more sections

Figures (21)

  • Figure 1: Topology-(un)aware aggregation for IID vs. OOD knowledge propagation. CIFAR10 is distributed across 64 nodes: OOD data placed on node with the fourth highest degree. Aggregation strategy for topology-unaware is Unweighted and topology-aware is Degree. Green indicates higher test accuracy on the respective dataset after 40 rounds of training; white indicates the opposite. Our proposed topology-aware method (right) achieves higher test OOD accuracies without sacrificing IID accuracies.
  • Figure 2: IID vs. OOD knowledge propagation. Data distributed in each topology as described in Appendix \ref{['appendix:iid_data_details']}, with OOD data located on the node with the fourth highest degree in the respective topology. We report average percent difference in test accuracy AUC between IID and OOD data over 40 rounds of training across all devices in a topology; averaged again over all topologies and seeds. Lower percent difference indicates that the OOD data did not propagate to as many nodes as the IID data.
  • Figure 3: Visual comparison of a single node's topology-unaware (Unweighted) vs. topology-aware (Degree, Betweenness) aggregation coefficients. Neighboring nodes colored and sized by their aggregation coefficients determined via the aggregation strategy. Gray nodes are not involved in aggregation for the aggregating node.
  • Figure 4: OOD knowledge propagation in three different decentralized topologies. In each case, OOD data are located on node with highest degree. Left to right: Experiments with MNIST, FMNIST, TinyMem, CIFAR10, CIFAR100. Green indicates node with OOD data. We vary the aggregation strategies: FL, Weighted, Unweighted, Random, Betweenness ($\tau=$ 0.1), Degree ($\tau=$ 0.1). Illustrated topologies shown for a single seed, while all bar plot results are averaged over three seeds.
  • Figure 5: Impact of OOD data location on OOD data spread. OOD data location is varied across the four highest degree nodes in each topology (we successively place the OOD data on nodes with lower degree). Left to right: Experiments on MNIST, FMNIST, TinyMem, CIFAR10, CIFAR100. We vary the aggregation strategies: Betweenness ($\tau=$ 0.1), Degree ($\tau=$ 0.1). Illustrated topologies shown for a single seed, while all bar plot results are averaged over three seeds.
  • ...and 16 more figures

Theorems & Definitions (2)

  • Definition B.1: Image Backdoor
  • Definition B.2: Language Backdoor