Towards understanding evolution of science through language model series

TL;DR

This work addresses how scientific discourse evolves by training a sequence of language models organized by publication year. The authors introduce AnnualBERT, a whole-word tokenization, year-by-year continual pretraining framework that starts from a base model trained on arXiv papers up to 2008 and then incrementally updates for each subsequent year. They demonstrate competitive performance on standard scientific NLP tasks, achieve state-of-the-art results on arXiv-specific link prediction, and use probing and weight-trajectory analyses to reveal learn-ing and forgetting dynamics that are often task-dependent. Additionally, they show that model weights follow a smooth temporal path and that interpolating between models across time can approximate the current-year model, offering insights into temporal adaptation and potential time-aware model editing. Overall, AnnualBERT provides a practical, interpretable approach to capturing the evolution of science in language representations with implications for domain-specific NLP and bibliometric analyses.

Abstract

We introduce AnnualBERT, a series of language models designed specifically to capture the temporal evolution of scientific text. Deviating from the prevailing paradigms of subword tokenizations and "one model to rule them all", AnnualBERT adopts whole words as tokens and is composed of a base RoBERTa model pretrained from scratch on the full-text of 1.7 million arXiv papers published until 2008 and a collection of progressively trained models on arXiv papers at an annual basis. We demonstrate the effectiveness of AnnualBERT models by showing that they not only have comparable performances in standard tasks but also achieve state-of-the-art performances on domain-specific NLP tasks as well as link prediction tasks in the arXiv citation network. We then utilize probing tasks to quantify the models' behavior in terms of representation learning and forgetting as time progresses. Our approach enables the pretrained models to not only improve performances on scientific text processing tasks but also to provide insights into the development of scientific discourse over time. The series of the models is available at https://huggingface.co/jd445/AnnualBERTs.

Figures (15)

• Figure 1: Workflow for developing our AnnualBERT models pretrained on arXiv papers. We first train from scratch a RoBERTa model on papers published until 2008, denoted as $\mathcal{M}_{\text{base}}$, and then for each subsequent year $t$ from 2009, we use papers published in that year for continual training of $\mathcal{M}_{t-1}$ to obtain $\mathcal{M}_{t}$. During the training process, we use whole word token, represented as $\text{WT}_1$, $\text{WT}_2$, …, $\text{WT}_n$.
• Figure 2: Summary statistics of our arXiv corpus. (a) Yearly number of papers by category. (b) Yearly number of sentences and tokens after cleaning.
• Figure 3: Jaccard similarity matrix between vocabularies.
• Figure 4: Experimental results for link prediction in the temporal arXiv citation network.
• Figure 5: The probabilities of top tokens given by different models, for the sentence shown at the top.