NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction

TL;DR

NABench tackles the need for fair, scalable evaluation of nucleotide foundation models for fitness prediction by assembling 162 DMS/SELEX assays and 2.6M mutated sequences into a unified benchmark. It systematically evaluates 29 NFMs under zero-shot, few-shot, supervised, and transfer settings, revealing substantial heterogeneity in performance across nucleic acid types and tasks. The results show that no single model excels across all scenarios: Evo-family models shine in zero-shot settings while BERT-like models benefit most from labeled data, and transfer learning offers additional gains, especially in cross-task contexts. By providing standardized data processing, evaluation protocols, and rich metadata, NABench enables robust baselines and accelerated progress in nucleic acid design, synthetic biology, and biochemistry; the associated codebase is publicly available for reproducibility.

Abstract

Nucleotide sequence variation can induce significant shifts in functional fitness. Recent nucleotide foundation models promise to predict such fitness effects directly from sequence, yet heterogeneous datasets and inconsistent preprocessing make it difficult to compare methods fairly across DNA and RNA families. Here we introduce NABench, a large-scale, systematic benchmark for nucleic acid fitness prediction. NABench aggregates 162 high-throughput assays and curates 2.6 million mutated sequences spanning diverse DNA and RNA families, with standardized splits and rich metadata. We show that NABench surpasses prior nucleotide fitness benchmarks in scale, diversity, and data quality. Under a unified evaluation suite, we rigorously assess 29 representative foundation models across zero-shot, few-shot prediction, transfer learning, and supervised settings. The results quantify performance heterogeneity across tasks and nucleic-acid types, demonstrating clear strengths and failure modes for different modeling choices and establishing strong, reproducible baselines. We release NABench to advance nucleic acid modeling, supporting downstream applications in RNA/DNA design, synthetic biology, and biochemistry. Our code is available at https://github.com/mrzzmrzz/NABench.

Figures (11)

• Figure 1: NABench provides a comprehensive framework for nucleic acid fitness prediction.
• Figure 2: Results of zero-shot tasks: (a). Distribution of correlation scores of models on every assay, colored by architecture. GPT-like GenerRNA and Evo models perform the best. (b). Ranking by three metrics namely Spearman R, AUC and MCC. The mean scores are used for the ranking. Top-5 models are colored in purple.
• Figure 3: Results of zero-shot tasks. (a) Distribution of $\rho$ across different types of nucleic sequences, with the top-5 models highlighted in purple. (b) Inference time versus parameter size for selected models. Transformer-based models are shown as grey points, while Evo-1.5, which incorporates Hyena blocks, is highlighted in orange. The number of parameters is plotted on a logarithmic scale. (c) $\rho$ versus parameter size for all models, where different architectures are represented by distinct marker shapes. The number of parameters is plotted on a logarithmic scale.
• Figure 4: Supervised learning: Supervised learning results for some top models, spearman's $\rho$ is reported.
• Figure 5: Benchmarking on SELEX data: Top performance models in few-shot and supervised learning.