GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models

TL;DR

GenBench addresses the lack of a standardized evaluation framework for Genomic Foundation Models by providing a modular benchmarking suite that spans coding, non-coding, and genome-architecture tasks across short- and long-range sequences. It evaluates ten GFMs, split into attention-based and convolution-based categories, over 43 datasets within a unified codebase to analyze architecture–data interactions, sequence length effects, and computational cost. The study finds that attention-based models excel on short-range tasks, while convolution-based models gain efficiency and approach similar performance on long-range tasks, with longer context generally boosting performance. These insights offer practical guidance for designing and training GFMs with reproducible benchmarks in genomics, facilitating fair comparisons and future methodological advances.

Abstract

The Genomic Foundation Model (GFM) paradigm is expected to facilitate the extraction of generalizable representations from massive genomic data, thereby enabling their application across a spectrum of downstream applications. Despite advancements, a lack of evaluation framework makes it difficult to ensure equitable assessment due to experimental settings, model intricacy, benchmark datasets, and reproducibility challenges. In the absence of standardization, comparative analyses risk becoming biased and unreliable. To surmount this impasse, we introduce GenBench, a comprehensive benchmarking suite specifically tailored for evaluating the efficacy of Genomic Foundation Models. GenBench offers a modular and expandable framework that encapsulates a variety of state-of-the-art methodologies. Through systematic evaluations of datasets spanning diverse biological domains with a particular emphasis on both short-range and long-range genomic tasks, firstly including the three most important DNA tasks covering Coding Region, Non-Coding Region, Genome Structure, etc. Moreover, We provide a nuanced analysis of the interplay between model architecture and dataset characteristics on task-specific performance. Our findings reveal an interesting observation: independent of the number of parameters, the discernible difference in preference between the attention-based and convolution-based models on short- and long-range tasks may provide insights into the future design of GFM.

Figures (9)

• Figure 1: The image illustrates the intricate organization of eukaryotic genomic DNA, highlighting three critical components: the Coding Region, the Non-coding Region, and Genome Architecture. The Coding Region, comprising only about 1.5% of the genome, is responsible for coding proteins, essential for cellular functions. In contrast, the vast Non-coding Region, making up 98.5% of the genome, plays crucial roles in gene regulation and genome stability, containing regulatory elements like promoters, enhancers, and silencers, as well as structural components. Genome Architecture is depicted by the DNA being tightly packed around histone proteins into chromosomes, ensuring efficient storage and regulation of genetic information. Notably, long-range interactions within the DNA are essential for life processes, facilitating the regulation of gene expression by bringing distant regulatory elements into proximity with target genes.
• Figure 2: Impact of length and parameter size on accuracy: (a) Accuracy variation across different lengths in species. (b) Accuracy variation across different lengths in promoter prediction tasks. (c) Evaluation of NT accuracy on short-range tasks with parameter sizes of 50M, 100M, and 500M.
• Figure 3: The t-SNE visualization of DNA embedding for foundation model in species classification. Including embedding for DNABERT2 with an accuracy of 0.742, embedding for HyenaDNA with an accuracy of 0.655, embedding for GENA-LM with an accuracy of 0.747, embedding for the NT with an accuracy of 0.761, and embedding for Caduceus with an accuracy of 0.703.
• Figure 4: Flops versus input length
• Figure 5: The graphical overview of GenBench.