Revisiting Convolution Architecture in the Realm of DNA Foundation Models

TL;DR

Revisiting Convolution Architecture in the Realm of DNA Foundation Models re-evaluates CNNs for DNA foundation modeling, proposing ConvNova—a CNN with dilated and gated convolutions and a dual-branch gating structure. Through extensive experiments on short- and long-range benchmarks (NT, Genomic Benchmarks, BEND gene finding, chromatin profile prediction), ConvNova achieves state-of-the-art results on many tasks, often with fewer parameters and faster computation than Transformer/SSM-based baselines. The work reveals that CNNs retain strong inductive biases for DNA sequences, highlights the importance of receptive-field design, and offers design insights such as the superiority of dilation over downsampling for genomic data. It also discusses biological implications of locality in histone marks and outlines limitations, including lack of multi-species pretraining and task scope, suggesting directions for future research. Overall, ConvNova demonstrates that CNN-based architectures can remain competitive and even surpass modern foundation models in DNA sequence modeling, prompting renewed interest in CNNs for genomics.

Abstract

In recent years, a variety of methods based on Transformer and state space model (SSM) architectures have been proposed, advancing foundational DNA language models. However, there is a lack of comparison between these recent approaches and the classical architecture convolutional networks (CNNs) on foundation model benchmarks. This raises the question: are CNNs truly being surpassed by these recent approaches based on transformer and SSM architectures? In this paper, we develop a simple but well-designed CNN-based method termed ConvNova. ConvNova identifies and proposes three effective designs: 1) dilated convolutions, 2) gated convolutions, and 3) a dual-branch framework for gating mechanisms. Through extensive empirical experiments, we demonstrate that ConvNova significantly outperforms recent methods on more than half of the tasks across several foundation model benchmarks. For example, in histone-related tasks, ConvNova exceeds the second-best method by an average of 5.8%, while generally utilizing fewer parameters and enabling faster computation. In addition, the experiments observed findings that may be related to biological characteristics. This indicates that CNNs are still a strong competitor compared to Transformers and SSMs. We anticipate that this work will spark renewed interest in CNN-based methods for DNA foundation models.

Figures (6)

• Figure 1: A) The trade-off between model size and accuracy (AUC score) of various methods. ConvNova achieves the current SoTA performance. B) The trade-off between input sequence length and runtime of different methods. ConvNova exhibits clear superiority over HyenaDNA and Caduceus. All models are around 7M parameters, tested on A100 80GB with batch size 1.
• Figure 2: ConvNova architecture. The ConvNova model processes double stands as inputs. The sequences are initially subjected to one-hot encoding and subsequently pass through a convolution layer. The processed data then enters a series of Gated Convolution Blocks (GCBs), the specifics of which are elaborated in § \ref{['gatecnn']}. The output from the GCBs is then fed into an MLP. The final stage of the framework bifurcates into two distinct heads: the downstream head and the pretraining head.
• Figure 3: Chromatin Profile Prediction results. AUC Score (↑) is reported for ConvNova, HyenaDNA, NT v2, DNABERT-2, and DeepATT performance on transcription factors (TF), DNase I hypersensitive sites (DHS), histone modifications (HM), and average score.
• Figure 4: The impact of dilation rate and GCB counts on the MCC in ConvNova model.
• Figure 5: Visualization of the attention maps for NTv2* (Left) and NTv2 (Right). The asterisk (*) denotes the modified version. The modified initialization places more emphasis on neighboring tokens in the attention map.