Federated Evaluation of On-device Personalization

TL;DR

Federated learning enables training global models without sending raw data to central servers, but personalization introduces risks of degrading some users' experiences. This work extends FL with Federated Personalization Evaluation (FPE), a privacy-preserving framework that splits local data into train and test, computes baseline and personalized metrics, and aggregates delta metrics to guide deployment decisions. In experiments on a next-word prediction model, personalization yielded meaningful gains across a large user base, with a mean relative improvement around 14.5% and substantial per-user gains under certain hyperparameters (e.g., about 47% of users achieving at least 0.02 improvement). The approach provides scalable guidance for hyperparameter tuning and gating in live inference, enabling privacy-preserving, large-scale evaluation of personalization strategies.

Abstract

Federated learning is a distributed, on-device computation framework that enables training global models without exporting sensitive user data to servers. In this work, we describe methods to extend the federation framework to evaluate strategies for personalization of global models. We present tools to analyze the effects of personalization and evaluate conditions under which personalization yields desirable models. We report on our experiments personalizing a language model for a virtual keyboard for smartphones with a population of tens of millions of users. We show that a significant fraction of users benefit from personalization.

Figures (3)

• Figure 1: An illustration of Federated Personalization Evaluation: (A) the global model (gray circle) is sent to client devices, (B) the device computes SGD updates on the train partition of the local data, resulting in a personalized model (the green square), (C) the device computes a report of metrics for the global and personalized model on the test partition of the local data, (D) pairs of metric reports are sent by various devices to the server, (E) the server computes histograms of various delta metrics.
• Figure 2: Accuracy delta histograms for different learning rates (L) and batch sizes (B): (a) At $B=5$ and $L=0.1$, 47% of users achieve $\geq$ 0.02 accuracy improvement; (b) At $B=10$ and $L=1.0$, 39% of users achieve $\geq$ 0.02 accuracy improvement; (c) At $B=10$ and $L=0.1$, 29% of users achieve $\geq$ 0.02 accuracy improvement.
• Figure 3: Analysis of accuracy deltas by learning rate (L) and batch size (B) sliced by (a) number of user tokens and (b) baseline accuracy.