Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DNAZEN: Enhanced Gene Sequence Representations via Mixed Granularities of Coding Units

Lei Mao, Yuanhe Tian, Yan Song

TL;DR

DNAZEN addresses the challenge of encoding gene sequences with multi-scale structure by introducing G-grams—long contiguous units extracted from large genomic corpora via PMI—into a Transformer-based genomic model. It deploys a dual-encoder architecture (G-gram encoder and E4BU for small units) and fuses their representations using a matching matrix, augmented by whole G-gram masking to train across granularities. Empirical results on the GUE benchmark show DNAZEN outperforms strong baselines across many tasks, with notable gains when pre-trained on GUE data and when leveraging more G-grams; visualizations and a case study demonstrate that G-grams capture species- and dataset-specific motifs and improve interpretability. The approach advances genome modeling by explicitly incorporating large-scale coding units, enabling better long-range dependence modeling and motif discovery with practical downstream impact.

Abstract

Genome modeling conventionally treats gene sequence as a language, reflecting its structured motifs and long-range dependencies analogous to linguistic units and organization principles such as words and syntax. Recent studies utilize advanced neural networks, ranging from convolutional and recurrent models to Transformer-based models, to capture contextual information of gene sequence, with the primary goal of obtaining effective gene sequence representations and thus enhance the models' understanding of various running gene samples. However, these approaches often directly apply language modeling techniques to gene sequences and do not fully consider the intrinsic information organization in them, where they do not consider how units at different granularities contribute to representation. In this paper, we propose DNAZEN, an enhanced genomic representation framework designed to learn from various granularities in gene sequences, including small polymers and G-grams that are combinations of several contiguous polymers. Specifically, we extract the G-grams from large-scale genomic corpora through an unsupervised approach to construct the G-gram vocabulary, which is used to provide G-grams in the learning process of DNA sequences through dynamically matching from running gene samples. A Transformer-based G-gram encoder is also proposed and the matched G-grams are fed into it to compute their representations and integrated into the encoder for basic unit (E4BU), which is responsible for encoding small units and maintaining the learning and inference process. To further enhance the learning process, we propose whole G-gram masking to train DNAZEN, where the model largely favors the selection of each entire G-gram to mask rather than an ordinary masking mechanism performed on basic units. Experiments on benchmark datasets demonstrate the effectiveness of DNAZEN on various downstream tasks.

DNAZEN: Enhanced Gene Sequence Representations via Mixed Granularities of Coding Units

TL;DR

DNAZEN addresses the challenge of encoding gene sequences with multi-scale structure by introducing G-grams—long contiguous units extracted from large genomic corpora via PMI—into a Transformer-based genomic model. It deploys a dual-encoder architecture (G-gram encoder and E4BU for small units) and fuses their representations using a matching matrix, augmented by whole G-gram masking to train across granularities. Empirical results on the GUE benchmark show DNAZEN outperforms strong baselines across many tasks, with notable gains when pre-trained on GUE data and when leveraging more G-grams; visualizations and a case study demonstrate that G-grams capture species- and dataset-specific motifs and improve interpretability. The approach advances genome modeling by explicitly incorporating large-scale coding units, enabling better long-range dependence modeling and motif discovery with practical downstream impact.

Abstract

Genome modeling conventionally treats gene sequence as a language, reflecting its structured motifs and long-range dependencies analogous to linguistic units and organization principles such as words and syntax. Recent studies utilize advanced neural networks, ranging from convolutional and recurrent models to Transformer-based models, to capture contextual information of gene sequence, with the primary goal of obtaining effective gene sequence representations and thus enhance the models' understanding of various running gene samples. However, these approaches often directly apply language modeling techniques to gene sequences and do not fully consider the intrinsic information organization in them, where they do not consider how units at different granularities contribute to representation. In this paper, we propose DNAZEN, an enhanced genomic representation framework designed to learn from various granularities in gene sequences, including small polymers and G-grams that are combinations of several contiguous polymers. Specifically, we extract the G-grams from large-scale genomic corpora through an unsupervised approach to construct the G-gram vocabulary, which is used to provide G-grams in the learning process of DNA sequences through dynamically matching from running gene samples. A Transformer-based G-gram encoder is also proposed and the matched G-grams are fed into it to compute their representations and integrated into the encoder for basic unit (E4BU), which is responsible for encoding small units and maintaining the learning and inference process. To further enhance the learning process, we propose whole G-gram masking to train DNAZEN, where the model largely favors the selection of each entire G-gram to mask rather than an ordinary masking mechanism performed on basic units. Experiments on benchmark datasets demonstrate the effectiveness of DNAZEN on various downstream tasks.
Paper Structure (19 sections, 4 equations, 3 figures, 5 tables)

This paper contains 19 sections, 4 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. The DNAZEN
  3. G-gram Construction
  4. G-gram Encoding
  5. Representation Enhancement with G-grams
  6. Training and Inference with G-grams
  7. Experiments
  8. Datasets and Tasks
  9. Implementation Details
  10. Results and Analysis
  11. Overall Results
  12. Effect of Model Initialization Approach
  13. Effect of Whole G-gram Masking
  14. Visualization of G-gram Distributions
  15. Case Study
  16. ...and 4 more sections

Figures (3)

  • Figure 1: The overview of DNAZEN. The left part presents the G-gram construction process, where the G-grams are obtained from large-scale gene corpora through an unsupervised approach named pointwise mutual information (PMI). The constructed G-gram vocabulary, the extracted G-grams, the matching matrix between the input tokens and the G-grams, as well as the G-gram encoder are shown in the middle of the figure. The original and tokenized input gene sequence, as well as the E4BU to enhance the gene representation with G-grams and to perform masked language modeling, are presented at the right part of the figure. The middle-right part is the whole G-gram masking used in pre-training to learn information of different granularities.
  • Figure 2: The G-gram distribution in different datasets. Figure (a) presents the distribution on the transcription factor prediction tasks for human and mouse; Figure (b) shows the distribution on ten datasets of epigenetic marks prediction. For both figures, the y-axis denotes different G-grams (which are not presented for better visualization), and the length of the bar and the color intensity represent each G-gram’s frequency divided by the total number of G-grams in that dataset.
  • Figure 3: Three cases to illustrate the effect of G-grams in helping DNAZEN to make correct predictions on the promoter detection task (the ones with TATA boxes). A part of the input DNA sequence is presented on the top, and the bottom shows some G-grams extracted from the input sequence. The G-grams with TATA box are highlighted by the dashed orange boxes.