Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DOLFIN: Balancing Stability and Plasticity in Federated Continual Learning

Omayma Moussadek, Riccardo Salami, Simone Calderara

TL;DR

DOLFIN tackles federated continual learning by combining a Vision Transformer backbone with task-specific, low-rank LoRA adapters and DualGradient Projection Memory to balance plasticity and stability under privacy constraints. The method updates only a small trainable subspace per task while enforcing orthogonality to past tasks, enabling interference-free learning without rehearsal. Across four benchmarks (CIFAR-100, ImageNet-R, ImageNet-A, CUB-200) and two Dirichlet heterogeneity settings, DOLFIN achieves state-of-the-art final average accuracy while maintaining comparable memory footprints to baselines, illustrating the practicality of orthogonal low-rank adapters in privacy-preserving FCIL. The approach reduces communication and avoids storing past data, making it suitable for scalable deployment in distributed settings where data cannot be centralized.

Abstract

Federated continual learning (FCL) enables models to learn new tasks across multiple distributed clients, protecting privacy and without forgetting previously acquired knowledge. However, current methods face challenges balancing performance, privacy preservation, and communication efficiency. We introduce a Distributed Online LoRA for Federated INcremental learning method DOLFIN, a novel approach combining Vision Transformers with low-rank adapters designed to efficiently and stably learn new tasks in federated environments. Our method leverages LoRA for minimal communication overhead and incorporates DualGradient Projection Memory (DualGPM) to prevent forgetting. Evaluated on CIFAR-100, ImageNet-R, ImageNet-A, and CUB-200 under two Dirichlet heterogeneity settings, DOLFIN consistently surpasses six strong baselines in final average accuracy while matching their memory footprint. Orthogonal low-rank adapters offer an effective and scalable solution for privacy-preserving continual learning in federated settings.

DOLFIN: Balancing Stability and Plasticity in Federated Continual Learning

TL;DR

DOLFIN tackles federated continual learning by combining a Vision Transformer backbone with task-specific, low-rank LoRA adapters and DualGradient Projection Memory to balance plasticity and stability under privacy constraints. The method updates only a small trainable subspace per task while enforcing orthogonality to past tasks, enabling interference-free learning without rehearsal. Across four benchmarks (CIFAR-100, ImageNet-R, ImageNet-A, CUB-200) and two Dirichlet heterogeneity settings, DOLFIN achieves state-of-the-art final average accuracy while maintaining comparable memory footprints to baselines, illustrating the practicality of orthogonal low-rank adapters in privacy-preserving FCIL. The approach reduces communication and avoids storing past data, making it suitable for scalable deployment in distributed settings where data cannot be centralized.

Abstract

Federated continual learning (FCL) enables models to learn new tasks across multiple distributed clients, protecting privacy and without forgetting previously acquired knowledge. However, current methods face challenges balancing performance, privacy preservation, and communication efficiency. We introduce a Distributed Online LoRA for Federated INcremental learning method DOLFIN, a novel approach combining Vision Transformers with low-rank adapters designed to efficiently and stably learn new tasks in federated environments. Our method leverages LoRA for minimal communication overhead and incorporates DualGradient Projection Memory (DualGPM) to prevent forgetting. Evaluated on CIFAR-100, ImageNet-R, ImageNet-A, and CUB-200 under two Dirichlet heterogeneity settings, DOLFIN consistently surpasses six strong baselines in final average accuracy while matching their memory footprint. Orthogonal low-rank adapters offer an effective and scalable solution for privacy-preserving continual learning in federated settings.
Paper Structure (14 sections, 4 equations, 1 figure, 2 tables)

This paper contains 14 sections, 4 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Learning.
  4. Class-Incremental Learning.
  5. Federated CIL
  6. Methodology
  7. Experiments
  8. Datasets and Preprocessing
  9. Evaluated approaches
  10. Implementation Details
  11. Hyperparameter Selection
  12. Evaluation Metrics.
  13. Results.
  14. Conclusion

Figures (1)

  • Figure 1: At each task, a new matrix $A_t$ is designed and kept frozen. Only the corresponding matrix $B_t$ is updated during training. All previous parameters, including earlier $B$ matrices and pre-trained weights, remain frozen.