BEACON: Benchmark for Comprehensive RNA Tasks and Language Models

TL;DR

BEACON addresses the absence of a standardized RNA benchmark by compiling 13 diverse tasks across Structure, Function, and Engineering and evaluating a spectrum of models from naive encoders to RNA foundation models. It further investigates RNA LM design by ablations on tokenization and positional encoding, concluding that single-nucleotide tokenization with ALiBi yields robust, data-efficient performance, leading to BEACON-B, an efficient baseline. The work provides a unified evaluation framework, extensive model analyses, and practical guidance for building resource-efficient RNA models, with datasets and code publicly accessible. Overall, BEACON clarifies the strengths and gaps of current RNA representations, guiding future research toward scalable, interpretable, and broadly applicable RNA modeling.

Abstract

RNA plays a pivotal role in translating genetic instructions into functional outcomes, underscoring its importance in biological processes and disease mechanisms. Despite the emergence of numerous deep learning approaches for RNA, particularly universal RNA language models, there remains a significant lack of standardized benchmarks to assess the effectiveness of these methods. In this study, we introduce the first comprehensive RNA benchmark BEACON (\textbf{BE}nchm\textbf{A}rk for \textbf{CO}mprehensive R\textbf{N}A Task and Language Models). First, BEACON comprises 13 distinct tasks derived from extensive previous work covering structural analysis, functional studies, and engineering applications, enabling a comprehensive assessment of the performance of methods on various RNA understanding tasks. Second, we examine a range of models, including traditional approaches like CNNs, as well as advanced RNA foundation models based on language models, offering valuable insights into the task-specific performances of these models. Third, we investigate the vital RNA language model components from the tokenizer and positional encoding aspects. Notably, our findings emphasize the superiority of single nucleotide tokenization and the effectiveness of Attention with Linear Biases (ALiBi) over traditional positional encoding methods. Based on these insights, a simple yet strong baseline called BEACON-B is proposed, which can achieve outstanding performance with limited data and computational resources. The datasets and source code of our benchmark are available at https://github.com/terry-r123/RNABenchmark.

Figures (2)

• Figure 1: Overview of BEACON. a: Categorization of the 13 benchmark tasks into classification and regression at both nucleotide and sequence levels. b: Diverse database distinguished by data size and source type. c: Visual representations of tasks across Structure, Function, and Engineering. d: List of baseline models, including naive supervised deep models and advanced RNA language models. e: Metrics for evaluating model performance in classification and regression tasks, tailored to RNA analysis specifics.
• Figure 2: Derivation of nucleotide-level representations. In single nucleotide tokenization, a token directly corresponds to a nucleotide, thus the representations are identical. For overlapping Kmer tokenization, the nucleotide representation is the averaged representation of tokens covering it. In Byte-Pair Encoding (BPE) and non-overlapping K-mer tokenization, the representation is derived from the token covering it.