DecHW: Heterogeneous Decentralized Federated Learning Exploiting Second-Order Information

TL;DR

This work tackles data and model initialization heterogeneity in Decentralized Federated Learning by introducing DecHW, a Hessian-based, parameter-wise aggregation method. By normalizing and accumulating the diagonal of the Hessian, DecHW assigns per-parameter weights to neighbor updates, enabling robust, topology-agnostic fusion without a central server. Empirical results on MNIST, Fashion-MNIST, and CIFAR-10 demonstrate faster convergence and higher accuracy than multiple baselines across varying non-IID data distributions, with a practical discussion on communication overhead. The approach offers a scalable, communication-efficient solution for decentralized learning in heterogeneous networks, with public code and clear advantages over existing KD or averaging-based methods.

Abstract

Decentralized Federated Learning (DFL) is a serverless collaborative machine learning paradigm where devices collaborate directly with neighbouring devices to exchange model information for learning a generalized model. However, variations in individual experiences and different levels of device interactions lead to data and model initialization heterogeneities across devices. Such heterogeneities leave variations in local model parameters across devices that leads to slower convergence. This paper tackles the data and model heterogeneity by explicitly addressing the parameter level varying evidential credence across local models. A novel aggregation approach is introduced that captures these parameter variations in local models and performs robust aggregation of neighbourhood local updates. Specifically, consensus weights are generated via approximation of second-order information of local models on their local datasets. These weights are utilized to scale neighbourhood updates before aggregating them into global neighbourhood representation. In extensive experiments with computer vision tasks, the proposed approach shows strong generalizability of local models at reduced communication costs.

Figures (9)

• Figure 1: Average accuracy across devices on the CIFAR10 dataset in DFL settings (details in Sec. \ref{['sec:exp_setup']}). DecHomo: Parameter-wise averaging with uniform model initialization. DecHetero: Parameter-wise averaging with diverse model initializations. DecHW: Proposed method with diverse model initializations.
• Figure 2: The norm of the Hessian diagonal was observed for a few nodes during training on the MNIST dataset, as outlined in Sec. \ref{['sec:exp_setup']}. As training progresses, the norms gradually decrease towards zero.
• Figure 3: Average accuracy and loss across devices on CIFAR10 dataset. Mean values from 5 different runs are reported as a function of communication rounds.
• Figure 4: DecHW is compared with its variant DecHW$^*$ on the CIFAR10 dataset.
• Figure 5: Data allocation across devices using Dirichlet distributions. The size of each circle indicates the quantity of data allocated from each class to a node.