Impact of network topology on the performance of Decentralized Federated Learning

TL;DR

This work investigates how network topology shapes learning dynamics in fully decentralized Federated Learning (DFL) at the edge. It analyzes Erdős-Rényi, Barabási-Albert, and Stochastic Block Model topologies under six data-distribution schemes using a DecAvg-style aggregation, on MNIST with a three-layer MLP. The key finding is that global centrality metrics (e.g., degree, betweenness) robustly correlate with final performance, while local clustering is a weak predictor; diffusion is hampered by dilution during aggregation, and central nodes exert a pull that accelerates knowledge spread. The results reveal topology-driven diffusion barriers and hub-driven amplification, with notable differences between intra- and inter-community knowledge transfer, offering practical guidance for topology-aware DFL design and data assignment strategies. The analysis also situates the ER graphs near the connectivity threshold $p^*=rac{\ln(N)}{N}$ to contextualize diffusion dynamics.

Abstract

Fully decentralized learning is gaining momentum for training AI models at the Internet's edge, addressing infrastructure challenges and privacy concerns. In a decentralized machine learning system, data is distributed across multiple nodes, with each node training a local model based on its respective dataset. The local models are then shared and combined to form a global model capable of making accurate predictions on new data. Our exploration focuses on how different types of network structures influence the spreading of knowledge - the process by which nodes incorporate insights gained from learning patterns in data available on other nodes across the network. Specifically, this study investigates the intricate interplay between network structure and learning performance using three network topologies and six data distribution methods. These methods consider different vertex properties, including degree centrality, betweenness centrality, and clustering coefficient, along with whether nodes exhibit high or low values of these metrics. Our findings underscore the significance of global centrality metrics (degree, betweenness) in correlating with learning performance, while local clustering proves less predictive. We highlight the challenges in transferring knowledge from peripheral to central nodes, attributed to a dilution effect during model aggregation. Additionally, we observe that central nodes exert a pull effect, facilitating the spread of knowledge. In examining degree distribution, hubs in Barabasi-Albert networks positively impact learning for central nodes but exacerbate dilution when knowledge originates from peripheral nodes. Finally, we demonstrate the formidable challenge of knowledge circulation outside of segregated communities.

Figures (33)

• Figure 1: The three BA networks considered in our experiments, with increasing value of the preferential attachment parameter $m$. Vertices are colored based on their degree, going from purple to red with increasing degree values.
• Figure 2: The three different realizations of the Erdős-Rényi graph, idempotent to the corresponding BA graph with increasing value of the preferential attachment parameter $m$. Vertices are colored based on their degree, going from purple to red with increasing degree values.
• Figure 3: The two SBM graphs. Colouring is based on the data distribution as described in the body of the paper.
• Figure 4: Degree distributions for the analyzed networks. From left to right: Barabási-Albert, Erdős-Rényi and Stochastic Block model.
• Figure 5: Scatterplots of the nodes' betweenness centrality measures vs the degree in our ER graphs. Increasing value of connectedness going from left to right. Red dots are the nodes chosen for the highest-focused case, the blue dots are the nodes chosen for the lowest-focused case. The scatterplots show that the betweenness centrality is proportional to the degree.