MetagenBERT: a Transformer-based Architecture using Foundational genomic Large Language Models for novel Metagenome Representation

TL;DR

MetagenBERT tackles the bottleneck of annotation-dependent metagenome representations by directly embedding raw reads with genomic LLMs and then aggregating these embeddings into fixed-length metagenome vectors via a global FAISS-accelerated K-Means clustering framework. The approach yields annotation-free cluster-abundance representations that, alone or concatenated with species abundances, achieve competitive disease-prediction performance across five gut-related datasets and demonstrate transferability through a MetaCardis-based foundation-model variant. Key contributions include demonstrating end-to-end metagenome embeddings from raw reads, evaluating read embedders DNABERT-2 and DNABERT-MS, and showing that a large, diverse pretraining corpus can support cross-cohort generalization. The work highlights substantial computational costs but also substantial gains in scalability and robustness, pointing toward practical foundation-model-like representations for heterogeneous metagenomic data across sequencing technologies.

Abstract

Metagenomic disease prediction commonly relies on species abundance tables derived from large, incomplete reference catalogs, constraining resolution and discarding valuable information contained in DNA reads. To overcome these limitations, we introduce MetagenBERT, a Transformer based framework that produces end to end metagenome embeddings directly from raw DNA sequences, without taxonomic or functional annotations. Reads are embedded using foundational genomic language models (DNABERT2 and the microbiome specialized DNABERTMS), then aggregated through a scalable clustering strategy based on FAISS accelerated KMeans. Each metagenome is represented as a cluster abundance vector summarizing the distribution of its embedded reads. We evaluate this approach on five benchmark gut microbiome datasets (Cirrhosis, T2D, Obesity, IBD, CRC). MetagenBERT achieves competitive or superior AUC performance relative to species abundance baselines across most tasks. Concatenating both representations further improves prediction, demonstrating complementarity between taxonomic and embedding derived signals. Clustering remains robust when applied to as little as 10% of reads, highlighting substantial redundancy in metagenomes and enabling major computational gains. We additionally introduce MetagenBERT Glob Mcardis, a cross cohort variant trained on the large, phenotypically diverse MetaCardis cohort and transferred to other datasets, retaining predictive signal including for unseen phenotypes, indicating the feasibility of a foundation model for metagenome representation. Robustness analyses (PERMANOVA, PERMDISP, entropy) show consistent separation of different states across subsamples. Overall, MetagenBERT provides a scalable, annotation free representation of metagenomes pointing toward future phenotype aware generalization across heterogeneous cohorts and sequencing technologies.

Figures (10)

• Figure 1: MetagenBERT-Glob: The Global Clustering Architecture.Panel (a). For each metagenome, all reads are embedded through the use of a gLLM. A subsample of each embedded metagenome is used to train a KMeans common to all the selected dataset. The centroids of the clusters obtained through K-means are retrieved, effectively partitioning the latent space. Panel (b). Every read embedding of each sample is then assigned to its cluster by nearest neighbor search against the centroids, thus creating a new abundance vector based on embeddings rather than species used for classification.
• Figure 2: Evaluation of DNABERT-MS against DNABERT-2 with intrinsic and extrinsic validation. On the left panel is the cross-entropy of DNABERT-MS for each checkpoint computed on real world evaluation datasets. The cross-entropy was calculated using DNABERT-2 for step 0, then the different checkpoints of DNABERT-MS on benchmark gut-related diseases datasets. We show only the first checkpoints for easy visualization. The cross-entropy is decreasing up to the 15.000 step, then the model appears to diverge due to overfitting. On the right panel is balanced accuracy of Phylum classification of real world sequences, showing consistent amelioration when using DNABERT-MS embeddings.
• Figure 3: Performances of MetagenBERT-Glob on 5 benchmark datasets with various numbers of clusters and embedders. MetagenBERT-Glob competes with a range of state-of-the-art methods across all datasets and tends to outperform them on T2D. In most configurations, although not universally, increasing the number of clusters generally leads to higher AUC. “MetagenBERT-Abu” refers to the performance of a LASSO model trained such as the one we used with MetagenBERT embeddings, but trained solely on species abundances. Across all five datasets, MetagenBERT-Glob either matches or surpasses the performance of the species-abundance-only model. When species abundances are concatenated with cluster abundances, performance generally improves, supporting the hypothesis that the two types of information are complementary. An exception is observed in the IBD dataset, where MetagenBERT-Glob performs better without the addition of species abundance. It is difficult to conclude that DNABERT-MS consistently outperforms DNABERT-2, as both embedders produce very similar results, with the notable exception of T2D, where MetagenBERT-MS performs unexpectedly poorly. The margins reported correspond to standard errors.
• Figure 4: Comparison of Prediction performance when using MetagenBERT-Glob trained directly on each benchmark dataset or when using cluster abundance obtained through Metacardis trained K-means. The figure shows that, even if the performances are globally lower than when clustering on the dataset of origin, this dataset efficiently captures important dynamics even with Metacardis K-means alone.
• Figure 5: Comparison of Prediction performance when using MetagenBERT-Glob with different proportions of reads assigned. The left panel of the figure shows performances heatmap according to different proportions of reads and number of clusters. Performances are comparable when using 10% and 100% of the data and start decreasing, especially for large number of clusters, under 3% of reads and are unsatisfying under 0.5% of reads. This results are confirmed on the right panel with the linear mixed model displaying mean performance with the evolution of reads proportion.