Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Task-Uniform Convergence and Backward Transfer in Federated Domain-Incremental Learning with Partial Participation

Longtao Xu, Jian Li

TL;DR

The paper tackles FDIL under partial participation by introducing SPECIAL, a memory-free server-side proximal augmentation to FedAvg that regularizes updates toward the previous global model. It proves two core results: a backward knowledge transfer bound under partial participation and the first task-uniform convergence rate across all tasks for FDIL with partial participation, achieving $O\left(\sqrt{\frac{E}{NT}}\right)$ under appropriate step sizes. Empirically, SPECIAL attains state-of-the-art ACC and competitive BWT across four domain-shift datasets without data replay or extra memory, demonstrating strong stability-plasticity trade-offs in realistic federated continual learning. The approach preserves communication and model size while enabling efficient, provable learning across evolving domains.

Abstract

Real-world federated systems seldom operate on static data: input distributions drift while privacy rules forbid raw-data sharing. We study this setting as Federated Domain-Incremental Learning (FDIL), where (i) clients are heterogeneous, (ii) tasks arrive sequentially with shifting domains, yet (iii) the label space remains fixed. Two theoretical pillars remain missing for FDIL under realistic deployment: a guarantee of backward knowledge transfer (BKT) and a convergence rate that holds across the sequence of all tasks with partial participation. We introduce SPECIAL (Server-Proximal Efficient Continual Aggregation for Learning), a simple, memory-free FDIL algorithm that adds a single server-side ``anchor'' to vanilla FedAvg: in each round, the server nudges the uniformly sampled participated clients update toward the previous global model with a lightweight proximal term. This anchor curbs cumulative drift without replay buffers, synthetic data, or task-specific heads, keeping communication and model size unchanged. Our theory shows that SPECIAL (i) preserves earlier tasks: a BKT bound caps any increase in prior-task loss by a drift-controlled term that shrinks with more rounds, local epochs, and participating clients; and (ii) learns efficiently across all tasks: the first communication-efficient non-convex convergence rate for FDIL with partial participation, O((E/NT)^(1/2)), with E local epochs, T communication rounds, and N participated clients per round, matching single-task FedAvg while explicitly separating optimization variance from inter-task drift. Experimental results further demonstrate the effectiveness of SPECIAL.

Task-Uniform Convergence and Backward Transfer in Federated Domain-Incremental Learning with Partial Participation

TL;DR

The paper tackles FDIL under partial participation by introducing SPECIAL, a memory-free server-side proximal augmentation to FedAvg that regularizes updates toward the previous global model. It proves two core results: a backward knowledge transfer bound under partial participation and the first task-uniform convergence rate across all tasks for FDIL with partial participation, achieving under appropriate step sizes. Empirically, SPECIAL attains state-of-the-art ACC and competitive BWT across four domain-shift datasets without data replay or extra memory, demonstrating strong stability-plasticity trade-offs in realistic federated continual learning. The approach preserves communication and model size while enabling efficient, provable learning across evolving domains.

Abstract

Real-world federated systems seldom operate on static data: input distributions drift while privacy rules forbid raw-data sharing. We study this setting as Federated Domain-Incremental Learning (FDIL), where (i) clients are heterogeneous, (ii) tasks arrive sequentially with shifting domains, yet (iii) the label space remains fixed. Two theoretical pillars remain missing for FDIL under realistic deployment: a guarantee of backward knowledge transfer (BKT) and a convergence rate that holds across the sequence of all tasks with partial participation. We introduce SPECIAL (Server-Proximal Efficient Continual Aggregation for Learning), a simple, memory-free FDIL algorithm that adds a single server-side ``anchor'' to vanilla FedAvg: in each round, the server nudges the uniformly sampled participated clients update toward the previous global model with a lightweight proximal term. This anchor curbs cumulative drift without replay buffers, synthetic data, or task-specific heads, keeping communication and model size unchanged. Our theory shows that SPECIAL (i) preserves earlier tasks: a BKT bound caps any increase in prior-task loss by a drift-controlled term that shrinks with more rounds, local epochs, and participating clients; and (ii) learns efficiently across all tasks: the first communication-efficient non-convex convergence rate for FDIL with partial participation, O((E/NT)^(1/2)), with E local epochs, T communication rounds, and N participated clients per round, matching single-task FedAvg while explicitly separating optimization variance from inter-task drift. Experimental results further demonstrate the effectiveness of SPECIAL.
Paper Structure (26 sections, 15 theorems, 69 equations, 5 figures, 3 tables, 1 algorithm)

This paper contains 26 sections, 15 theorems, 69 equations, 5 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Federated Domain-Incremental Learning
  4. Client- vs. Server-Side Proximal Terms
  5. The SPECIAL Algorithm
  6. Theoretical Analysis
  7. Assumption
  8. Backward Knowledge Transfer Analysis
  9. Convergence Across All Tasks
  10. Intuitions and Proof Sketch
  11. Experiments
  12. Related work
  13. Continual Learning
  14. Federated Continual Learning
  15. Federated Domain-Incremental Learning
  16. ...and 11 more sections

Key Result

Theorem 1

Let $\theta_K^{t}$ denote the global model after round $t$ of task $K$, and set $\theta_K^{0}=\theta_{K-1}$. In each round $\tau$, the server samples a subset $\mathcal{S}_K^\tau\subseteq[M]$ of size $N$ uniformly without replacement and aggregates only those clients. Suppose that for every $\tau\in where the expectation is over client sampling and data stochasticity.

Figures (5)

  • Figure 1: Illustration of FDIL on the PACS dataset. In addition to distribute and aggregate models, the server retains information from the previous task $i-1$, either as a small exemplar buffer $\mathcal{D}_{i-1}$ or, more compactly, as the preceding global model $\theta_{i-1}$.
  • Figure 2: Test accuracy on Task $1$, Task $2$, and their union for Digit-10 under client-side vs. server-side proximal terms.
  • Figure 3: The SPECIAL Algorithm
  • Figure 4: Relation between ACC and BWT under varying values of $\lambda$. Results are demonstrated on three datasets: Digit-10 (circles), VLCS (diamonds), PACS (stars), and DN4IL (triangles), and the color of each point represents the value of $\lambda$ from $0$ to $1$.
  • Figure 5: Visualized performance w.r.t $T$, $E$, $\alpha$.

Theorems & Definitions (26)

  • Theorem 1: BKT under partial participation
  • Remark 1: Positioning to prior BKT theory
  • Lemma 1: Uniform within-task drift
  • Theorem 2: Task-uniform convergence of SPECIAL
  • Remark 2: Novelty and relation to prior analyses
  • Corollary 1
  • Remark 3: Trade-off between computation and communication
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • ...and 16 more