Improving Language Plasticity via Pretraining with Active Forgetting

TL;DR

This work tackles the challenge of rapidly adapting pretrained language models (PLMs) to new languages by introducing an active forgetting mechanism during pretraining. By periodically resetting the token embedding layer (and optimizer states), the PLM experiences episodic learning that increases its linguistic plasticity, enabling faster and more data-efficient rewiring to unseen languages. Empirical results with RoBERTa-base show that forgetting-pretrained models achieve large gains in low-data cross-lingual transfer tasks (XNLI, MLQA, XQuAD) and converge faster during adaptation, with the largest benefits for languages distant from English. The findings suggest forgetting as a principled tool to enhance cross-language transfer and model reusability, with potential extension to other pretraining regimes and modalities.

Abstract

Pretrained language models (PLMs) are today the primary model for natural language processing. Despite their impressive downstream performance, it can be difficult to apply PLMs to new languages, a barrier to making their capabilities universally accessible. While prior work has shown it possible to address this issue by learning a new embedding layer for the new language, doing so is both data and compute inefficient. We propose to use an active forgetting mechanism during pretraining, as a simple way of creating PLMs that can quickly adapt to new languages. Concretely, by resetting the embedding layer every K updates during pretraining, we encourage the PLM to improve its ability of learning new embeddings within a limited number of updates, similar to a meta-learning effect. Experiments with RoBERTa show that models pretrained with our forgetting mechanism not only demonstrate faster convergence during language adaptation but also outperform standard ones in a low-data regime, particularly for languages that are distant from English.

Figures (12)

• Figure 1: Rewiring via relearning token embeddings: where the transformer body (the purple part) is "frozen" and reused for a new language, but the token embeddings are relearned to suit the new language.
• Figure 2: Unsupervised zero-shot cross-lingual transfer. Left: in the pretrain stage, we compare standard pretraining with forgetting pretraining, where the token embeddings are actively forgotten at a regular interval while the transformer body is learned as the standard pretraining. Middle: the task adapt stage and the language adapt stage separately adapt the transformer body using English task data and the token embeddings using unlabeled data in the new language. Right: the assemble stage reassemble the adapted body and token embedding layer into a usable PLM.
• Figure 3: The rewiring performance for standard PLMs (blue dashed line) drops drastically if the adaptation tokens $\leq 10\text{M}$.
• Figure 4: Pretraining loss curves of forgetting and standard language models. The forgetting mechanism brings an episodic pattern into the loss curve: every embedding forgetting produces a loss spike, from which the model learn to recover. Through such repeats of forget-relearn, the model gets used to learn new embeddings from scratch.
• Figure 5: Adaptation curves on XNLI, MLQA, and XQuAD. Numbers aggregated across languages. The first row contains the full adaptation curves, which comprises $125\text{K}$ adaptation steps. The second row contains the zoom-in versions of curves for the first $5\text{K}$ adaptation steps. Forgetting PLMs converge faster than standard PLMs; for instance, on XQuAD (the last plot), forgetting PLMs reach $92\%$ of their final performance within $5\text{K}$ updates, while standard PLMs only reached $53\%$ of their final performance at that point.