Vanishing Variance Problem in Fully Decentralized Neural-Network Systems
Yongding Tian, Zaid Al-Ars, Maksim Kitsak, Peter Hofstee
TL;DR
This work identifies a vanishing variance problem when averaging uncorrelated neural networks in fully decentralized gossip learning, which delays convergence. It proposes a variance-corrected model averaging method that layer-wise preserves the Xavier variances of input models, making gossip learning converge as efficiently as federated learning under both IID and non-IID data. The approach demonstrates substantial performance gains over standard gossip learning and competing variants, achieving up to 6x faster convergence in large-scale networks. The results imply that variance-aware aggregation can unlock practical, privacy-preserving decentralized learning with FL-like efficiency.
Abstract
Federated learning and gossip learning are emerging methodologies designed to mitigate data privacy concerns by retaining training data on client devices and exclusively sharing locally-trained machine learning (ML) models with others. The primary distinction between the two lies in their approach to model aggregation: federated learning employs a centralized parameter server, whereas gossip learning adopts a fully decentralized mechanism, enabling direct model exchanges among nodes. This decentralized nature often positions gossip learning as less efficient compared to federated learning. Both methodologies involve a critical step: computing a representation of received ML models and integrating this representation into the existing model. Conventionally, this representation is derived by averaging the received models, exemplified by the FedAVG algorithm. Our findings suggest that this averaging approach inherently introduces a potential delay in model convergence. We identify the underlying cause and refer to it as the "vanishing variance" problem, where averaging across uncorrelated ML models undermines the optimal variance established by the Xavier weight initialization. Unlike federated learning where the central server ensures model correlation, and unlike traditional gossip learning which circumvents this problem through model partitioning and sampling, our research introduces a variance-corrected model averaging algorithm. This novel algorithm preserves the optimal variance needed during model averaging, irrespective of network topology or non-IID data distributions. Our extensive simulation results demonstrate that our approach enables gossip learning to achieve convergence efficiency comparable to that of federated learning.