Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PlantBiMoE: A Bidirectional Foundation Model with SparseMoE for Plant Genomes

Kepeng Lin, Qizhe Zhang, Rui Wang, Xuehai Hu, Wei Xu

TL;DR

This work addresses the need for an efficient, bidirectional, long-context plant genome language model by introducing PlantBiMoE, which fuses a BiMamba-based bidirectional state-space core with a SparseMoE routing mechanism. Pretrained on 42 plant species (25.40B nucleic acid pairs) and evaluated on the MPGB benchmark (31 datasets, 11 tasks, 50–6,000 bp), PlantBiMoE achieves the best average performance on 9 of 11 tasks and 20 of 31 subdatasets, while using only 116M parameters with 64M active, and a 32,768 bp context window. The results demonstrate strong cross-species generalization and long-range dependency modeling with improved efficiency, outperforming AgroNT and PDLLMs on most tasks. The study highlights the importance of architectural design and data quality in genomic language modeling and provides a reproducible benchmark (MPGB) and codebase for future work.

Abstract

Understanding the underlying linguistic rules of plant genomes remains a fundamental challenge in computational biology. Recent advances including AgroNT and PDLLMs have made notable progress although, they suffer from excessive parameter size and limited ability to model the bidirectional nature of DNA strands respectively. To address these limitations, we propose PlantBiMoE, a lightweight and expressive plant genome language model that integrates bidirectional Mamba and a Sparse Mixture-of-Experts (SparseMoE) framework. The bidirectional Mamba enables the model to effectively capture structural dependencies across both the forward and reverse DNA strands, while SparseMoE significantly reduces the number of active parameters, improving computational efficiency without sacrificing modeling capacity. We evaluated and tested our model on the Modified Plants Genome Benchmark (MPGB), an enhanced genomic benchmark, which consolidates 31 datasets across 11 representative tasks, with input sequence lengths ranging from 50 to 6,000 bp. Experimental results demonstrate that PlantBiMoE achieves the best performance on 20 out of 31 datasets and the average best when comparing with existing models. In summary, all above results demonstrate that our model can effectively represent plant genomic sequences, serving as a robust computational tool for diverse genomic tasks, while making substantive contributions to plant genomics, gene editing, and synthetic biology. The code is available at: https://github.com/HUST-Keep-Lin/PlantBiMoE

PlantBiMoE: A Bidirectional Foundation Model with SparseMoE for Plant Genomes

TL;DR

This work addresses the need for an efficient, bidirectional, long-context plant genome language model by introducing PlantBiMoE, which fuses a BiMamba-based bidirectional state-space core with a SparseMoE routing mechanism. Pretrained on 42 plant species (25.40B nucleic acid pairs) and evaluated on the MPGB benchmark (31 datasets, 11 tasks, 50–6,000 bp), PlantBiMoE achieves the best average performance on 9 of 11 tasks and 20 of 31 subdatasets, while using only 116M parameters with 64M active, and a 32,768 bp context window. The results demonstrate strong cross-species generalization and long-range dependency modeling with improved efficiency, outperforming AgroNT and PDLLMs on most tasks. The study highlights the importance of architectural design and data quality in genomic language modeling and provides a reproducible benchmark (MPGB) and codebase for future work.

Abstract

Understanding the underlying linguistic rules of plant genomes remains a fundamental challenge in computational biology. Recent advances including AgroNT and PDLLMs have made notable progress although, they suffer from excessive parameter size and limited ability to model the bidirectional nature of DNA strands respectively. To address these limitations, we propose PlantBiMoE, a lightweight and expressive plant genome language model that integrates bidirectional Mamba and a Sparse Mixture-of-Experts (SparseMoE) framework. The bidirectional Mamba enables the model to effectively capture structural dependencies across both the forward and reverse DNA strands, while SparseMoE significantly reduces the number of active parameters, improving computational efficiency without sacrificing modeling capacity. We evaluated and tested our model on the Modified Plants Genome Benchmark (MPGB), an enhanced genomic benchmark, which consolidates 31 datasets across 11 representative tasks, with input sequence lengths ranging from 50 to 6,000 bp. Experimental results demonstrate that PlantBiMoE achieves the best performance on 20 out of 31 datasets and the average best when comparing with existing models. In summary, all above results demonstrate that our model can effectively represent plant genomic sequences, serving as a robust computational tool for diverse genomic tasks, while making substantive contributions to plant genomics, gene editing, and synthetic biology. The code is available at: https://github.com/HUST-Keep-Lin/PlantBiMoE

Paper Structure

This paper contains 10 sections, 6 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. MATERIALS AND METHODS
  3. Dataset
  4. Model Architecture
  5. Results
  6. Model Training And Setup
  7. Baseline
  8. Metrics
  9. Results on MPGB
  10. Discussion and Conclusion

Figures (4)

  • Figure 1: Model architecture of PlantBiMoE.
  • Figure 2: Pre-training species categories and size for PlantBiMoE.
  • Figure 3: PlantBiMoE predicts Polyadenylation, Splicing site, LncRNA and Enhancer region. (a)Receiver operating characteristic curve for Polyadenylation prediction . (b) Same as in a, but for Splicing site prediction. (c) Same as in a, but for LncRNA. (d) Same as in a, but for Enhancer region.
  • Figure 4: Box plot of Chromatin accessibility task for six species.