CoPeP: Benchmarking Continual Pretraining for Protein Language Models

TL;DR

The Continual Pretraining of Protein Language Models (CoPeP) benchmark is introduced, a novel benchmark for evaluating continual learning approaches on pLMs and reveals that incorporating temporal meta-information improves perplexity by up to 7% even when compared to training on data from all tasks jointly.

Abstract

Protein language models (pLMs) have recently gained significant attention for their ability to uncover relationships between sequence, structure, and function from evolutionary statistics, thereby accelerating therapeutic drug discovery. These models learn from large protein databases that are continuously updated by the biology community and whose dynamic nature motivates the application of continual learning, not only to keep up with the ever-growing data, but also as an opportunity to take advantage of the temporal meta-information that is created during this process. As a result, we introduce the Continual Pretraining of Protein Language Models (CoPeP) benchmark, a novel benchmark for evaluating continual learning approaches on pLMs. Specifically, we curate a sequence of protein datasets derived from the UniProt Knowledgebase spanning a decade and define metrics to assess pLM performance across 31 protein understanding tasks. We evaluate several methods from the continual learning literature, including replay, unlearning, and plasticity-based methods, some of which have never been applied to models and data of this scale. Our findings reveal that incorporating temporal meta-information improves perplexity by up to 7% even when compared to training on data from all tasks jointly. Moreover, even at scale, several continual learning methods outperform naive continual pretraining. The CoPeP benchmark offers an exciting opportunity to study these methods at scale in an impactful real-world application.

Figures (25)

• Figure 1: Overview of the CoPeP benchmark. The UniProt Knowledgebase is continuously updated by biologists, which reflects the continuous discovery and curation of proteins. For every year's task, we pull the latest UniRef100 release derived from that year's UniProtKB release and use it to update the model. We evaluate several continual learning approaches on this sequence of datasets and assess their performance on a variety of protein understanding tasks.
• Figure 2: Temporal evolution of UniRef100 from 2015 to 2024. The database shows a steady upward trend in total sequences despite millions of sequences being culled annually.
• Figure 3: Left: Evolution of sequence identity distributions relative to the validation set. We track the sequence identity of the UniRef100 entries in each year compared to their nearest neighbor in the validation set. The plot shows the change in sequence identity density relative to the 2015 baseline. A systematic shift toward lower identity values (peak at $\sim$0.25) indicates that newer sequences are increasingly divergent from the validation set. Right: Year-over-year density shifts. The initial divergence is driven by substantial shifts in early years, whereas later transitions (2020--2024) show smaller, stabilizing fluctuations.
• Figure 4: Impact of temporal data filtering. Models are trained exclusively on the intersection of sequences retained between two UniRef100 releases. Left shows the validation perplexity for each model and right shows the corresponding dataset sizes. The best performance in each row is boxed in red, and the best performance in each column is boxed in green. Most years show improved validation perplexity on at least one intersection dataset despite a substantial reduction in dataset size. The best performance by any model is achieved by a model trained on the intersection of 2022 and 2024, despite only having 64% of the data of the larger 2024 release.
• Figure 5: Perplexity (left) and Sequence Recovery (right) on the UniProt validation set with shading representing standard error. Continual learning baselines generally improve over time, and outperform jointly training on all data. Temporal Replay achieves the best performance outperforming other continual methods and single year matched training for all years. Every continual baseline outperforms the naive continual learning baseline, validating many of them for the first time at scale.