Look Back for More: Harnessing Historical Sequential Updates for Personalized Federated Adapter Tuning

TL;DR

pFedSeq tackles the challenge of data heterogeneity in personalized federated learning by exploiting historical sequences of client adapter updates. It introduces a server-side Sequential Learner, implemented with Selective State Space Models, to generate per-client calibrations that customize a globally aggregated adapter for fine-tuning large foundation models via adapters like LoRA. The framework combines standard FedAvg-based aggregation with a hypernetwork-based sequence model to capture cross-client and cross-step relations, and it demonstrates superior performance over ten state-of-the-art PFL methods on four benchmarks (CIFAR-100, Tiny-ImageNet, DomainNet, Omniglot). The results show significant gains, particularly in domain-shifted and real-world heterogeneous settings, highlighting the practical impact of leveraging historical updates for robust personalization in federated adapter tuning.

Abstract

Personalized federated learning (PFL) studies effective model personalization to address the data heterogeneity issue among clients in traditional federated learning (FL). Existing PFL approaches mainly generate personalized models by relying solely on the clients' latest updated models while ignoring their previous updates, which may result in suboptimal personalized model learning. To bridge this gap, we propose a novel framework termed pFedSeq, designed for personalizing adapters to fine-tune a foundation model in FL. In pFedSeq, the server maintains and trains a sequential learner, which processes a sequence of past adapter updates from clients and generates calibrations for personalized adapters. To effectively capture the cross-client and cross-step relations hidden in previous updates and generate high-performing personalized adapters, pFedSeq adopts the powerful selective state space model (SSM) as the architecture of sequential learner. Through extensive experiments on four public benchmark datasets, we demonstrate the superiority of pFedSeq over state-of-the-art PFL methods.

Figures (12)

• Figure 1: Comparison between (a) existing PFL methods and (b) our approach. Instead of leveraging only the latest updates, our approach accounts for past learning trajectories by modeling cross-client and cross-step relations in previous steps, providing a broader view for identifying the consistent trends for learning more robust personalized adapters.
• Figure 2: An overview of pFedSeq framework. At each communication round $t$, clients perform local adapter tuning and send the adapter updates $\{\Delta_i^t\}_{i=1}^N$ to server. At the server, the updated adapters are aggregated to form a global adapter $\tilde{\theta}_g^t$. Meanwhile, the sequential learner processes the sequence of updates collected at the server and generates personalized calibrations, which are then applied to the global adapter to produce the personalized adapters $\{\theta_i^t\}_{i=1}^N$ and send to the clients.
• Figure 3: (a) Optimization and inference processes of sequential learner at the $t$-th round. (b) An instantiation of sequential learner using Selective SSM.
• Figure 4: Learning curves of pFedSeq and compared baselines for CIFAR-100 and Tiny-ImageNet.
• Figure 5: Performance of pFedSeq by varying the maximum sequence length $L$ on Omniglot.