OmniGenBench: Automating Large-scale in-silico Benchmarking for Genomic Foundation Models
Heng Yang, Jack Cole, Ke Li
TL;DR
OmniGenBench tackles the lack of standardized, scalable benchmarking for genomic foundation models by integrating four large-scale benchmarks ($42$ million sequences from $75$ datasets) into an open-source AutoBench framework, complemented by a public online hub and leaderboard. It standardizes benchmark suites with metadata, fixed hyperparameters, and common metrics, while providing wrapper interfaces to accommodate diverse GFMs and tokenizers. Empirical results across RGB, PGB, GUE, and GB show OmniGenome—an RNA/DNA-capable GFM—achieving top-tier or competitive performance, with clear benefits from incorporating RNA structure context and cross-modal training. The work highlights the importance of standardized evaluation, cross-species generalization, and community collaboration, while noting limitations such as task-specific gains and the need for broader in-vivo validation and protocol harmonization.
Abstract
The advancements in artificial intelligence in recent years, such as Large Language Models (LLMs), have fueled expectations for breakthroughs in genomic foundation models (GFMs). The code of nature, hidden in diverse genomes since the very beginning of life's evolution, holds immense potential for impacting humans and ecosystems through genome modeling. Recent breakthroughs in GFMs, such as Evo, have attracted significant investment and attention to genomic modeling, as they address long-standing challenges and transform in-silico genomic studies into automated, reliable, and efficient paradigms. In the context of this flourishing era of consecutive technological revolutions in genomics, GFM studies face two major challenges: the lack of GFM benchmarking tools and the absence of open-source software for diverse genomics. These challenges hinder the rapid evolution of GFMs and their wide application in tasks such as understanding and synthesizing genomes, problems that have persisted for decades. To address these challenges, we introduce GFMBench, a framework dedicated to GFM-oriented benchmarking. GFMBench standardizes benchmark suites and automates benchmarking for a wide range of open-source GFMs. It integrates millions of genomic sequences across hundreds of genomic tasks from four large-scale benchmarks, democratizing GFMs for a wide range of in-silico genomic applications. Additionally, GFMBench is released as open-source software, offering user-friendly interfaces and diverse tutorials, applicable for AutoBench and complex tasks like RNA design and structure prediction. To facilitate further advancements in genome modeling, we have launched a public leaderboard showcasing the benchmark performance derived from AutoBench. GFMBench represents a step toward standardizing GFM benchmarking and democratizing GFM applications.