Character-level Tokenizations as Powerful Inductive Biases for RNA Foundational Models

TL;DR

ChaRNABERT is presented, a suite of sample and parameter-efficient RNA foundational models, that through a learnable tokenization process are able to reach state-of-the-art performance on several tasks in established benchmarks.

Abstract

RNA is a vital biomolecule with numerous roles and functions within cells, and interest in targeting it for therapeutic purposes has grown significantly in recent years. However, fully understanding and predicting RNA behavior, particularly for applications in drug discovery, remains a challenge due to the complexity of RNA structures and interactions. While foundational models in biology have demonstrated success in modeling several biomolecules, especially proteins, achieving similar breakthroughs for RNA has proven more difficult. Current RNA models have yet to match the performance observed in the protein domain, leaving an important gap in computational biology. In this work, we present ChaRNABERT, a suite of sample and parameter-efficient RNA foundational models, that through a learnable tokenization process, are able to reach state-of-the-art performance on several tasks in established benchmarks. We extend its testing in relevant downstream tasks such as RNA-protein and aptamer-protein interaction prediction. Weights and inference code for ChaRNABERT-8M will be provided for academic research use. The other models will be available upon request.

Figures (11)

• Figure 1: ChaRNABERT's architecture. We train our models utilizing two datasets: one with exclusively non-coding sequences and another that combines both coding and non-coding sequences. Through Gradient-Based Subsequence Tokenization (GBST), the model learns optimal tokenization patterns for RNA sequences. ChaRNABERT employs a standard BERT transformer encoder, accommodating an input context of up to 8,190 nucleotides, and is trained using a masked language modeling objective to capture sequence information.
• Figure 2: Performance comparison with different context windows. We present the final exponential moving average (EMA) loss results for two tokenisation methods, EM (dashed lines) and GBST (solid lines), evaluated in different model size configurations, which are plotted in logarithmic scale on the X-axis. Each colour indicates a different context window length (2048, 4096 and 8192).
• Figure 3: Performance comparison across data sizes. The figure displays the final EMA loss for two tokenization methods, EM (dashed lines) and GBST (solid lines), evaluated across various model sizes (plotted in logarithmic scale on the X-axis). Different colors represent distinct data sizes (15M, 66M, 100M, and 150M of sequences).
• Figure 4: FLOPs vs. training loss across model sizes. We show the relationship between computational cost (FLOPs, on a logarithmic scale, X-axis) and training loss (Y-axis) for models of varying sizes.
• Figure 5: Best performance achieved by CRB relative to the corresponding baseline for each downstream task.