Federated Learning for Emoji Prediction in a Mobile Keyboard

TL;DR

This work tackles emoji prediction in mobile keyboards under privacy constraints by deploying a CIFG-LSTM model trained with federated learning. It combines language-model pretraining, a dedicated triggering mechanism, and diversification to balance accuracy, latency, and emoji variety, achieving superior top-1 accuracy in federated evaluations and strong live-traffic gains. Deployments use low-latency inference with quantized TensorFlow Lite and demonstrate practical privacy-preserving training for real-world NLU tasks. Overall, the study demonstrates that federated learning can yield production-quality, on-device models despite sparse, imbalanced data and strict latency requirements.

Abstract

We show that a word-level recurrent neural network can predict emoji from text typed on a mobile keyboard. We demonstrate the usefulness of transfer learning for predicting emoji by pretraining the model using a language modeling task. We also propose mechanisms to trigger emoji and tune the diversity of candidates. The model is trained using a distributed on-device learning framework called federated learning. The federated model is shown to achieve better performance than a server-trained model. This work demonstrates the feasibility of using federated learning to train production-quality models for natural language understanding tasks while keeping users' data on their devices.

Figures (4)

• Figure 1: Emoji predictions in Gboard. Based on the context "This party is lit", Gboard predicts both emoji and words.
• Figure 2: Accuracy@1 vs. training step with and without pretraining, using server-based evaluations.
• Figure 3: Distribution of 15 most frequently used emoji in English (US).
• Figure 4: Evaluation Accuracy@1 vs. Round for federated and server trained models.