Closed-form merging of parameter-efficient modules for Federated Continual Learning

TL;DR

This work addresses merging parameter-efficient LoRA modules in Federated Class-Incremental Learning by introducing LoRM, an alternating optimization framework that yields a closed-form-like solution for merging low-rank adapters. It derives exact merging equations when sharing either the A or B factor, and implements a training scheme that alternates which factor to optimize, reducing communication and preserving privacy through Gram-based summaries. Empirically, LoRM achieves state-of-the-art performance across diverse vision datasets and demonstrates robustness to data heterogeneity and domain shifts, including out-of-domain tasks. The approach offers a practical, scalable path for modular, continual, and federated learning in real-world scenarios by enabling precise, efficient, and privacy-preserving integration of task-specific components.

Abstract

Model merging has emerged as a crucial technique in Deep Learning, enabling the integration of multiple models into a unified system while preserving perfor-mance and scalability. In this respect, the compositional properties of low-rank adaptation techniques (e.g., LoRA) have proven beneficial, as simple averaging LoRA modules yields a single model that mostly integrates the capabilities of all individual modules. Building on LoRA, we take a step further by imposing that the merged model matches the responses of all learned modules. Solving this objective in closed form yields an indeterminate system with A and B as unknown variables, indicating the existence of infinitely many closed-form solutions. To address this challenge, we introduce LoRM, an alternating optimization strategy that trains one LoRA matrix at a time. This allows solving for each unknown variable individually, thus finding a unique solution. We apply our proposed methodology to Federated Class-Incremental Learning (FCIL), ensuring alignment of model responses both between clients and across tasks. Our method demonstrates state-of-the-art performance across a range of FCIL scenarios. The code to reproduce our experiments is available at github.com/aimagelab/fed-mammoth.

Figures (3)

• Figure 1: Training and aggregation procedure of LoRM, alternating between rounds. The figure depicts both Federated and Incremental aggregation for a generic layer.
• Figure 2: Speed of convergence.
• Figure 3: Alternating optimization.