Fast and Low-Cost Genomic Foundation Models via Outlier Removal

TL;DR

The paper tackles the challenge of deploying genomic foundation models under limited compute by mitigating outlier effects in transformer attention. It introduces Germ, an outlier-free Hopfield-layer-based architecture that replaces vanilla attention, enabling quantization-friendly, fast adaptation, and Germ-T as a continual-learning variant to avoid full retraining. Empirically, Germ delivers substantial gains, with average improvements of $37.98\%$ in fine-tuning and $64.34\%$ in quantization over a baseline DNABERT-2, while dramatically reducing outlier metrics (average kurtosis by $92.14\%$ and max infinity norm by $82.77\%$) across 27 datasets; Germ-T also provides notable robustness, though with some trade-offs at very low-bit quantization. These results hold across multiple tasks and models, including NT-500M-human and NT-2.5B-multi, and translate into practical benefits for deployment on edge and CPU-only environments. The work thus offers a scalable, resource-efficient path for genomic modeling with durable quantization robustness and rapid adaptation, while acknowledging limitations of Germ-T in extreme quantization scenarios and outlining future directions for robust QAT strategies.

Abstract

To address the challenge of scarce computational resources in genomic modeling, we introduce GERM, a genomic foundation model with strong compression performance and fast adaptability. GERM improves upon models like DNABERT-2 by eliminating outliers that hinder low-rank adaptation and post-training quantization, enhancing both efficiency and robustness. We replace the vanilla attention layer with an outlier-free mechanism inspired by associative memory models. By removing outliers during both pre-training and fine-tuning, this approach accelerates adaptation, reduces computational costs, and enhances quantization robustness within acceptable loss margins. Additionally, we propose GERM-T, a strategy that employs small-step continual learning within the outlier-free framework, leveraging original checkpoints to avoid retraining from scratch. Empirically, GERM improves fine-tuning performance by 37.98% and quantization by 64.34% over the baseline model. It also reduces average kurtosis by 92.14% and maximum infinity norm by 82.77%. Compared to leading methods, GERM consistently delivers superior performance, offering a practical solution for genomic modeling in resource-constrained settings. Code is available at https://github.com/MAGICS-LAB/GERM.

Figures (3)

• Figure 1: Structural Comparison of DNABERT-2 and Germ Models. This diagram illustrates the differences in processing pipelines between DNABERT-2 and Germ. Both DNABERT-2 and Germ use the SciencePiece tokenizer with BPE for tokenization. Following that, both models employ ALiBi for positional encoding in the embedding layer. However, as shown in (a), DNABERT-2's transformer architecture outputs the outliers. We propose replacing the vanilla Softmax with an outlier-free layer. In (b), the output of the attention mechanism removes outliers from the original output.
• Figure 2: Comparison of Performance in Resource-Constrained Computing Environments. Comparison of three models on the quantization and fine-tuning task. All models were trained on the same computing infrastructure (Nvidia GeForce RTX 2080 TI 11GB) for fair comparison. The training time represents the average time per epoch, with OmniQuant used as quantization example in this figure.
• Figure 3: Attention Distribution Visualization in Germ. Comparison of attention probabilities and outputs for a genomic sample between DNABERT-2 and NT-500M-human. Heatmaps from the final hidden layers are scaled from 0 (blue) to 1 (red). In the figure, all rows labeled (a) correspond to the vanilla DNABERT-2, while all rows labeled (b) represent the Germ version. The vanilla model exhibits a broad attention spread, which dilutes focus across tokens, whereas Germ concentrates attention on key tokens, enhancing both efficiency and interpretability.

Theorems & Definitions (6)

• Theorem 1.1: Informal
• Definition 1.1: Definition of target model ${ \macc@depth1 \frozen@everymath{\mathgroup\macc@group} \macc@set@skewchar \macc@nested@a111{} }$ and adopted model $f$
• Definition 1.2: Best Low-rank Approximation for $W$
• Lemma 1.1: Exactly represent target model, Lemma 7 of zeng2023expressive
• Theorem 1.2: Express capability of transformers
• proof