Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BioMamba: A Pre-trained Biomedical Language Representation Model Leveraging Mamba

Ling Yue, Sixue Xing, Yingzhou Lu, Tianfan Fu

TL;DR

BioMamba addresses the need for domain-specific biomedical language understanding by extending the Mamba architecture with Structured State Space Models (SSMs) and dynamic parameters, enabling efficient long-sequence processing. Pretrained on PubMed abstracts and initialized from Mamba-130m, BioMamba is further fine-tuned on biomedical QA data (BioASQ) to achieve state-of-the-art performance. Empirical results show BioMamba significantly surpasses BioBERT, BioGPT, and general Mamba, including substantial reductions in perplexity and cross-entropy on benchmark datasets and PubMed text. The work highlights the importance of domain-focused pretraining for biomedical NLP and releases code and models to foster further research, with implications for literature mining and clinical decision support.

Abstract

The advancement of natural language processing (NLP) in biology hinges on models' ability to interpret intricate biomedical literature. Traditional models often struggle with the complex and domain-specific language in this field. In this paper, we present BioMamba, a pre-trained model specifically designed for biomedical text mining. BioMamba builds upon the Mamba architecture and is pre-trained on an extensive corpus of biomedical literature. Our empirical studies demonstrate that BioMamba significantly outperforms models like BioBERT and general-domain Mamba across various biomedical tasks. For instance, BioMamba achieves a 100 times reduction in perplexity and a 4 times reduction in cross-entropy loss on the BioASQ test set. We provide an overview of the model architecture, pre-training process, and fine-tuning techniques. Additionally, we release the code and trained model to facilitate further research.

BioMamba: A Pre-trained Biomedical Language Representation Model Leveraging Mamba

TL;DR

BioMamba addresses the need for domain-specific biomedical language understanding by extending the Mamba architecture with Structured State Space Models (SSMs) and dynamic parameters, enabling efficient long-sequence processing. Pretrained on PubMed abstracts and initialized from Mamba-130m, BioMamba is further fine-tuned on biomedical QA data (BioASQ) to achieve state-of-the-art performance. Empirical results show BioMamba significantly surpasses BioBERT, BioGPT, and general Mamba, including substantial reductions in perplexity and cross-entropy on benchmark datasets and PubMed text. The work highlights the importance of domain-focused pretraining for biomedical NLP and releases code and models to foster further research, with implications for literature mining and clinical decision support.

Abstract

The advancement of natural language processing (NLP) in biology hinges on models' ability to interpret intricate biomedical literature. Traditional models often struggle with the complex and domain-specific language in this field. In this paper, we present BioMamba, a pre-trained model specifically designed for biomedical text mining. BioMamba builds upon the Mamba architecture and is pre-trained on an extensive corpus of biomedical literature. Our empirical studies demonstrate that BioMamba significantly outperforms models like BioBERT and general-domain Mamba across various biomedical tasks. For instance, BioMamba achieves a 100 times reduction in perplexity and a 4 times reduction in cross-entropy loss on the BioASQ test set. We provide an overview of the model architecture, pre-training process, and fine-tuning techniques. Additionally, we release the code and trained model to facilitate further research.
Paper Structure (24 sections, 9 equations, 2 figures, 5 tables)

This paper contains 24 sections, 9 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Recurrent neural network (RNN).
  4. Transformer.
  5. Self-supervised learning and pretraining.
  6. BERT
  7. GPT
  8. Structured State Space for Sequences model.
  9. S4 versus Transformer.
  10. Biomedical NLP Models.
  11. Method
  12. Mamba Model
  13. Pre-training BioMamba
  14. Fine-tuning BioMamba
  15. Experiments
  16. ...and 9 more sections

Figures (2)

  • Figure 1: RNN versus Transformer.
  • Figure 2: (left) Comparison of BERT and Mamba. BERT is pretrained on large corpora using masked language modeling devlin2018bert, its bidirectional nature provides a deep understanding of the context within the text. Differently, Mamba is an autoregressive model (unidirectional) and is pretrained on unlabeled language corpus by predicting the next token gu2023mamba. (right) BioMamba: First, BioMamba uses the parameters of the general-purpose Mamba for initialization. Then, BioMamba is further pretrained on a biomedical text corpus (e.g., PubMed canese2013pubmed). After that, BioMamba can be fine-tuned on downstream tasks (e.g., the question-answering BioASQ dataset).