Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text

Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Roxana Daneshjou, Jonathan Frankle, Percy Liang, Michael Carbin, Christopher D. Manning

TL;DR

BioMedLM presents a 2.7B parameter GPT-style model trained exclusively on PubMed text to address privacy and cost concerns of giant LLMs in biomedicine. The model achieves competitive performance on multiple biomedical QA benchmarks (MedMCQA, MedQA, MMLU Medical Genetics, PubMedQA, BioASQ) and demonstrates useful free-form answers when fine-tuned for patient-facing queries. The authors emphasize a transparent, privacy-preserving, open-source alternative that can be fine-tuned on modest hardware and data-tracing is possible due to full documentation of training data. The work highlights that domain-specific pretraining can enable smaller models to reach near or competitive levels of performance relative to much larger models, supporting broader accessibility in biomedical NLP.

Abstract

Models such as GPT-4 and Med-PaLM 2 have demonstrated impressive performance on a wide variety of biomedical NLP tasks. However, these models have hundreds of billions of parameters, are computationally expensive to run, require users to send their input data over the internet, and are trained on unknown data sources. Can smaller, more targeted models compete? To address this question, we build and release BioMedLM, a 2.7 billion parameter GPT-style autoregressive model trained exclusively on PubMed abstracts and full articles. When fine-tuned, BioMedLM can produce strong multiple-choice biomedical question-answering results competitive with much larger models, such as achieving a score of 57.3% on MedMCQA (dev) and 69.0% on the MMLU Medical Genetics exam. BioMedLM can also be fine-tuned to produce useful answers to patient questions on medical topics. This demonstrates that smaller models can potentially serve as transparent, privacy-preserving, economical and environmentally friendly foundations for particular NLP applications, such as in biomedicine. The model is available on the Hugging Face Hub: https://huggingface.co/stanford-crfm/BioMedLM.

BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text

TL;DR

BioMedLM presents a 2.7B parameter GPT-style model trained exclusively on PubMed text to address privacy and cost concerns of giant LLMs in biomedicine. The model achieves competitive performance on multiple biomedical QA benchmarks (MedMCQA, MedQA, MMLU Medical Genetics, PubMedQA, BioASQ) and demonstrates useful free-form answers when fine-tuned for patient-facing queries. The authors emphasize a transparent, privacy-preserving, open-source alternative that can be fine-tuned on modest hardware and data-tracing is possible due to full documentation of training data. The work highlights that domain-specific pretraining can enable smaller models to reach near or competitive levels of performance relative to much larger models, supporting broader accessibility in biomedical NLP.

Abstract

Models such as GPT-4 and Med-PaLM 2 have demonstrated impressive performance on a wide variety of biomedical NLP tasks. However, these models have hundreds of billions of parameters, are computationally expensive to run, require users to send their input data over the internet, and are trained on unknown data sources. Can smaller, more targeted models compete? To address this question, we build and release BioMedLM, a 2.7 billion parameter GPT-style autoregressive model trained exclusively on PubMed abstracts and full articles. When fine-tuned, BioMedLM can produce strong multiple-choice biomedical question-answering results competitive with much larger models, such as achieving a score of 57.3% on MedMCQA (dev) and 69.0% on the MMLU Medical Genetics exam. BioMedLM can also be fine-tuned to produce useful answers to patient questions on medical topics. This demonstrates that smaller models can potentially serve as transparent, privacy-preserving, economical and environmentally friendly foundations for particular NLP applications, such as in biomedicine. The model is available on the Hugging Face Hub: https://huggingface.co/stanford-crfm/BioMedLM.
Paper Structure (39 sections, 2 figures, 17 tables)

This paper contains 39 sections, 2 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. GPT-Neo 2.7B.
  4. PubMedBERT.
  5. BioLinkBERT and DRAGON.
  6. Galactica.
  7. BioGPT.
  8. Flan-PaLM and Med-PaLM.
  9. ChatGPT and GPT-4.
  10. Fully Open Language Models And Datasets.
  11. Model Design And Training
  12. Model Architecture
  13. BioMedLM Tokenizer
  14. Pre-training
  15. Fine-tuning
  16. ...and 24 more sections

Figures (2)

  • Figure 1: Train and Validation Loss after 100k Batches
  • Figure 2: Comparison of GPT-Neo 2.7B and BioMedLM on Select QA Tasks