FedRewind: Rewinding Continual Model Exchange for Decentralized Federated Learning

TL;DR

The combination of federated and continual learning concepts enables the FedRewind method to tackle the more challenging federated continual learning task, with data shifts over both space and time, surpassing existing baselines.

Abstract

In this paper, we present FedRewind, a novel approach to decentralized federated learning that leverages model exchange among nodes to address the issue of data distribution shift. Drawing inspiration from continual learning (CL) principles and cognitive neuroscience theories for memory retention, FedRewind implements a decentralized routing mechanism where nodes send/receive models to/from other nodes in the federation to address spatial distribution challenges inherent in distributed learning (FL). During local training, federation nodes periodically send their models back (i.e., rewind) to the nodes they received them from for a limited number of iterations. This strategy reduces the distribution shift between nodes' data, leading to enhanced learning and generalization performance. We evaluate our method on multiple benchmarks, demonstrating its superiority over standard decentralized federated learning methods and those enforcing specific routing schemes within the federation. Furthermore, the combination of federated and continual learning concepts enables our method to tackle the more challenging federated continual learning task, with data shifts over both space and time, surpassing existing baselines.

Figures (3)

• Figure 1: Rewind strategy. The model received and trained on the current node is sent back to its source node for a brief fine-tuning. The model then returns to the node and continue its training before the start of a new federated round.
• Figure 2: Performance at different degrees of data heterogeneity ($\alpha_{dir}$) on CIFAR-10 for 10 (left) and 50 (right) nodes.
• Figure 3: Training trend of rewind strategy in AFCL