Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

mRNA2vec: mRNA Embedding with Language Model in the 5'UTR-CDS for mRNA Design

Honggen Zhang, Xiangrui Gao, June Zhang, Lipeng Lai

TL;DR

mRNA2vec tackles the design of functional mRNA sequences by unifying the 5' UTR and CDS into a single input and training a contextual, data2vec-inspired embedding. By employing region-aware hard masking and auxiliary pretext tasks for Minimum Free Energy and Secondary Structure, the approach yields improved translation-related predictions (Translation Efficiency and Expression Level) and competitive performance on mRNA stability and protein production. The method demonstrates state-of-the-art improvements on 5' UTR tasks and robust cross-dataset generalization for CDS tasks, with practical implications for efficient mRNA design in vaccines and therapeutics. Overall, mRNA2vec advances sequence representation learning in the mRNA domain and provides a scalable, domain-informed pretraining framework that can accelerate mRNA design pipelines.

Abstract

Messenger RNA (mRNA)-based vaccines are accelerating the discovery of new drugs and revolutionizing the pharmaceutical industry. However, selecting particular mRNA sequences for vaccines and therapeutics from extensive mRNA libraries is costly. Effective mRNA therapeutics require carefully designed sequences with optimized expression levels and stability. This paper proposes a novel contextual language model (LM)-based embedding method: mRNA2vec. In contrast to existing mRNA embedding approaches, our method is based on the self-supervised teacher-student learning framework of data2vec. We jointly use the 5' untranslated region (UTR) and coding sequence (CDS) region as the input sequences. We adapt our LM-based approach specifically to mRNA by 1) considering the importance of location on the mRNA sequence with probabilistic masking, 2) using Minimum Free Energy (MFE) prediction and Secondary Structure (SS) classification as additional pretext tasks. mRNA2vec demonstrates significant improvements in translation efficiency (TE) and expression level (EL) prediction tasks in UTR compared to SOTA methods such as UTR-LM. It also gives a competitive performance in mRNA stability and protein production level tasks in CDS such as CodonBERT.

mRNA2vec: mRNA Embedding with Language Model in the 5'UTR-CDS for mRNA Design

TL;DR

mRNA2vec tackles the design of functional mRNA sequences by unifying the 5' UTR and CDS into a single input and training a contextual, data2vec-inspired embedding. By employing region-aware hard masking and auxiliary pretext tasks for Minimum Free Energy and Secondary Structure, the approach yields improved translation-related predictions (Translation Efficiency and Expression Level) and competitive performance on mRNA stability and protein production. The method demonstrates state-of-the-art improvements on 5' UTR tasks and robust cross-dataset generalization for CDS tasks, with practical implications for efficient mRNA design in vaccines and therapeutics. Overall, mRNA2vec advances sequence representation learning in the mRNA domain and provides a scalable, domain-informed pretraining framework that can accelerate mRNA design pipelines.

Abstract

Messenger RNA (mRNA)-based vaccines are accelerating the discovery of new drugs and revolutionizing the pharmaceutical industry. However, selecting particular mRNA sequences for vaccines and therapeutics from extensive mRNA libraries is costly. Effective mRNA therapeutics require carefully designed sequences with optimized expression levels and stability. This paper proposes a novel contextual language model (LM)-based embedding method: mRNA2vec. In contrast to existing mRNA embedding approaches, our method is based on the self-supervised teacher-student learning framework of data2vec. We jointly use the 5' untranslated region (UTR) and coding sequence (CDS) region as the input sequences. We adapt our LM-based approach specifically to mRNA by 1) considering the importance of location on the mRNA sequence with probabilistic masking, 2) using Minimum Free Energy (MFE) prediction and Secondary Structure (SS) classification as additional pretext tasks. mRNA2vec demonstrates significant improvements in translation efficiency (TE) and expression level (EL) prediction tasks in UTR compared to SOTA methods such as UTR-LM. It also gives a competitive performance in mRNA stability and protein production level tasks in CDS such as CodonBERT.
Paper Structure (20 sections, 3 equations, 5 figures, 4 tables)

This paper contains 20 sections, 3 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. mRNA Sequence-based Translation Prediction
  4. mRNA Sequence Representation Learning
  5. Method:mRNA2vec
  6. mRNA Pre-training
  7. pretraining Architecture
  8. Auxiliary Pretext Tasks with MFE and SS
  9. mRNA Translation Downstream Tasks
  10. Experiments
  11. Dataset
  12. Experiment Setting
  13. Pre-training Result
  14. Results
  15. Discussion
  16. ...and 5 more sections

Figures (5)

  • Figure 1: The diagram for the pretraining. The data2vec teacher model parameter $\Delta$ will be updated by the current student model parameter $\theta$ using EMA. The input sequence will be masked based on the hard-mask distribution $q$. The $\phi$ and $\psi$ are the neural networks used for MFE regression and SS classification.
  • Figure 2: Evaluate pretraining strategies:The data2vec with T5 encoder, the T5 encoder, and the untrained model (Unloaded)
  • Figure 4: Compare our method with the methods working on CDS region
  • Figure 5: Two sub-sequence selection strategies. Pre-selected: change the input length of the encoder. Post-selected: change the output length of the encoder. The result will change as the sequence length changes.
  • Figure 6: Evaluate different pretrain strategies on 5$'$ UTR region