Personalized Federated Learning via Stacking

TL;DR

The work tackles non-IID data in federated learning by introducing a model-agnostic stacking approach for personalization, where clients publish privacy-preserving base models and train a personalized meta-model on private data. This scheme eliminates the need for a single global model while enabling straightforward contribution evaluation through feature-importances, applicable to horizontal, vertical, and hybrid federations. Empirical results across simulated heterogeneity scenarios show that training personalized meta-models on held-out private data often yields the largest gains, though benefits vary with data distribution and dataset characteristics. Overall, the method offers a scalable, flexible pathway to personalized FL with interpretable contribution signals for fair collaboration in privacy-aware settings.

Abstract

Traditional Federated Learning (FL) methods typically train a single global model collaboratively without exchanging raw data. In contrast, Personalized Federated Learning (PFL) techniques aim to create multiple models that are better tailored to individual clients' data. We present a novel personalization approach based on stacked generalization where clients directly send each other privacy-preserving models to be used as base models to train a meta-model on private data. Our approach is flexible, accommodating various privacy-preserving techniques and model types, and can be applied in horizontal, hybrid, and vertically partitioned federations. Additionally, it offers a natural mechanism for assessing each client's contribution to the federation. Through comprehensive evaluations across diverse simulated data heterogeneity scenarios, we showcase the effectiveness of our method.

Figures (6)

• Figure 1: A high level outline of the proposed personalization approach. A client shares a privacy-preserving (PP) model trained on private data along with metadata with the federation and trains a personalized stacked model with its own private model and those fetched from the federation. It can then use feature importance methods to rank or score the contributions of each retrieved model and report it to the federation. In our experiments each personalized models is evaluated on a held-out test set of each client's private data.
• Figure 2: Quantity skew results: Plots (a)-(c) show the balanced accuracy gain of personalized meta-models over private ones, trained on both held-out and pooled data. Plots (d)-(f) depict performance gain, importance, and self-importance relative to private data size. Note that (d) and (e) focus on meta-models stacked on held-out data.
• Figure 3: Label skew results. Plots (a)-(c) show balanced accuracy gain. Plots (d) and (e) depict performance gain and importance relative to clients' private label balance, respectively. Plot (f) displays assigned importance as a function of both a client's and another's private label balance. Note that (d)-(f) focus on meta-models stacked on held-out data.
• Figure 4: Vertical partitioning results. Plots (a)-(c) depict balanced accuracy gain based on the proportion $p$ of total features randomly assigned to clients and the noise level $\varepsilon$ added to default values. Plot (d) illustrates assigned importance as a function of Jaccard similarity between feature sets. Plots (e) and (f) show self-importance relative to varying $p$ and $\varepsilon$, for both meta-models stacked on held-out and pooled data. Note that plots (a)-(d) focus on meta-models stacked on held-out data.
• Figure 5: Natural partitioning results. Plot (a) displays the column used for dataset partitioning, along with the resulting number of clients and their proportion of total data. Plots (b) and (c) show the performance gain and self-importance of meta-models stacked on both held-out and pooled data across datasets.