Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Diminishing Returns of Masked Language Models to Science

Zhi Hong, Aswathy Ajith, Gregory Pauloski, Eamon Duede, Kyle Chard, Ian Foster

TL;DR

This paper tests whether the scaling laws observed for general masked language models extend to science-focused tasks by evaluating 14 BERT-based models, including ScholarBERT variants trained on a large, multidisciplinary scientific corpus. Through 12 downstream scientific information extraction tasks, it finds that larger models, more data, or longer pretraining/finetuning often yield only modest or inconsistent gains, with domain-specific pretraining helping primarily in Biomedical domains. The results suggest that finetuning can compensate for domain gaps in some cases and that very large pretraining data may saturate benefits for certain tasks. The work highlights practical implications for deploying scientific NLP systems and provides ScholarBERT resources for the community, while acknowledging limitations in evaluation datasets and the need to explore tasks beyond NER and relation extraction.

Abstract

Transformer-based masked language models such as BERT, trained on general corpora, have shown impressive performance on downstream tasks. It has also been demonstrated that the downstream task performance of such models can be improved by pretraining larger models for longer on more data. In this work, we empirically evaluate the extent to which these results extend to tasks in science. We use 14 domain-specific transformer-based models (including ScholarBERT, a new 770M-parameter science-focused masked language model pretrained on up to 225B tokens) to evaluate the impact of training data, model size, pretraining and finetuning time on 12 downstream scientific tasks. Interestingly, we find that increasing model sizes, training data, or compute time does not always lead to significant improvements (i.e., >1% F1), if at all, in scientific information extraction tasks and offered possible explanations for the surprising performance differences.

The Diminishing Returns of Masked Language Models to Science

TL;DR

This paper tests whether the scaling laws observed for general masked language models extend to science-focused tasks by evaluating 14 BERT-based models, including ScholarBERT variants trained on a large, multidisciplinary scientific corpus. Through 12 downstream scientific information extraction tasks, it finds that larger models, more data, or longer pretraining/finetuning often yield only modest or inconsistent gains, with domain-specific pretraining helping primarily in Biomedical domains. The results suggest that finetuning can compensate for domain gaps in some cases and that very large pretraining data may saturate benefits for certain tasks. The work highlights practical implications for deploying scientific NLP systems and provides ScholarBERT resources for the community, while acknowledging limitations in evaluation datasets and the need to explore tasks beyond NER and relation extraction.

Abstract

Transformer-based masked language models such as BERT, trained on general corpora, have shown impressive performance on downstream tasks. It has also been demonstrated that the downstream task performance of such models can be improved by pretraining larger models for longer on more data. In this work, we empirically evaluate the extent to which these results extend to tasks in science. We use 14 domain-specific transformer-based models (including ScholarBERT, a new 770M-parameter science-focused masked language model pretrained on up to 225B tokens) to evaluate the impact of training data, model size, pretraining and finetuning time on 12 downstream scientific tasks. Interestingly, we find that increasing model sizes, training data, or compute time does not always lead to significant improvements (i.e., >1% F1), if at all, in scientific information extraction tasks and offered possible explanations for the surprising performance differences.
Paper Structure (18 sections, 3 figures, 6 tables)

This paper contains 18 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data and Methodology
  4. The Public Resource Dataset
  5. Models
  6. ScholarBERT Pretraining
  7. Experimental Results
  8. Sensitivity Analysis
  9. Finetuning
  10. Discussion
  11. Conclusions
  12. Extant BERT-based models
  13. ScholarBERT Pretraining Details
  14. Tokenization
  15. RoBERTa Optimizations
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Pretraining loss plots for the ScholarBERT models listed in \ref{['tab:models']}. The vertical dashed lines indicate the approximate locations of the iteration checkpoints selected for evaluation in \ref{['sec:sensitivity']}.
  • Figure 2: NER F1 scores from \ref{['tab:token-cls-appendix']}, with standard deviations represented by error bars.
  • Figure 3: Sentence classification F1 scores from \ref{['tab:sent-cls-appendix']}, with standard deviations represented by error bars.