Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Absorb & Escape: Overcoming Single Model Limitations in Generating Genomic Sequences

Zehui Li, Yuhao Ni, Guoxuan Xia, William Beardall, Akashaditya Das, Guy-Bart Stan, Yiren Zhao

TL;DR

A post-training sampling method, termed Absorb&Escape (A&E) is proposed, to perform compositional generation from AR models and DMs in heterogeneous genomic sequence generation, pointing out crucial limitations in both methods.

Abstract

Abstract Recent advances in immunology and synthetic biology have accelerated the development of deep generative methods for DNA sequence design. Two dominant approaches in this field are AutoRegressive (AR) models and Diffusion Models (DMs). However, genomic sequences are functionally heterogeneous, consisting of multiple connected regions (e.g., Promoter Regions, Exons, and Introns) where elements within each region come from the same probability distribution, but the overall sequence is non-homogeneous. This heterogeneous nature presents challenges for a single model to accurately generate genomic sequences. In this paper, we analyze the properties of AR models and DMs in heterogeneous genomic sequence generation, pointing out crucial limitations in both methods: (i) AR models capture the underlying distribution of data by factorizing and learning the transition probability but fail to capture the global property of DNA sequences. (ii) DMs learn to recover the global distribution but tend to produce errors at the base pair level. To overcome the limitations of both approaches, we propose a post-training sampling method, termed Absorb & Escape (A&E) to perform compositional generation from AR models and DMs. This approach starts with samples generated by DMs and refines the sample quality using an AR model through the alternation of the Absorb and Escape steps. To assess the quality of generated sequences, we conduct extensive experiments on 15 species for conditional and unconditional DNA generation. The experiment results from motif distribution, diversity checks, and genome integration tests unequivocally show that A&E outperforms state-of-the-art AR models and DMs in genomic sequence generation.

Absorb & Escape: Overcoming Single Model Limitations in Generating Genomic Sequences

TL;DR

A post-training sampling method, termed Absorb&Escape (A&E) is proposed, to perform compositional generation from AR models and DMs in heterogeneous genomic sequence generation, pointing out crucial limitations in both methods.

Abstract

Abstract Recent advances in immunology and synthetic biology have accelerated the development of deep generative methods for DNA sequence design. Two dominant approaches in this field are AutoRegressive (AR) models and Diffusion Models (DMs). However, genomic sequences are functionally heterogeneous, consisting of multiple connected regions (e.g., Promoter Regions, Exons, and Introns) where elements within each region come from the same probability distribution, but the overall sequence is non-homogeneous. This heterogeneous nature presents challenges for a single model to accurately generate genomic sequences. In this paper, we analyze the properties of AR models and DMs in heterogeneous genomic sequence generation, pointing out crucial limitations in both methods: (i) AR models capture the underlying distribution of data by factorizing and learning the transition probability but fail to capture the global property of DNA sequences. (ii) DMs learn to recover the global distribution but tend to produce errors at the base pair level. To overcome the limitations of both approaches, we propose a post-training sampling method, termed Absorb & Escape (A&E) to perform compositional generation from AR models and DMs. This approach starts with samples generated by DMs and refines the sample quality using an AR model through the alternation of the Absorb and Escape steps. To assess the quality of generated sequences, we conduct extensive experiments on 15 species for conditional and unconditional DNA generation. The experiment results from motif distribution, diversity checks, and genome integration tests unequivocally show that A&E outperforms state-of-the-art AR models and DMs in genomic sequence generation.

Paper Structure

This paper contains 46 sections, 1 theorem, 10 equations, 22 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Limitations of Existing Single-Model Approaches in Generating Genomic Sequences
  3. Model Composition As a Solution
  4. Preliminaries and Related Work
  5. Problem Formulation: DNA Generation
  6. Homogeneous vs. Heterogeneous Sequences
  7. Single Model Limitations in Heterogeneous Sequence Generation
  8. A Toy Example
  9. Method
  10. The Absorb & Escape Framework
  11. Absorb-Step
  12. Escape-Step
  13. Practical Implementation: Fast A&E
  14. Experiment
  15. Transcription Profile (TP) conditioned Promoter Design
  16. ...and 31 more sections

Key Result

Proposition 1

The Absorb & Escape (A&E) algorithm converges to the target distribution $p^{C}_{\theta,\beta}(\mathbf{x}) = p^{AR}_\theta(\mathbf{x}) \circ p^{DM}_\beta(\mathbf{x})$, under the assumptions that both models are properly trained, the segments of $\mathbf{x}$ are homogeneous, the subset of segments is

Figures (22)

  • Figure 1: (a) Generated DNA interacting with TATA-binding protein. (b) Proposed A&E framework.
  • Figure 2: A toy example with heterogeneous sequences: (a) The overall training set consists of $N=50,000$ heterogeneous sequences, where each sequence further consists of 16 homogeneous segments. We apply an autoregressive and a diffusion model to learn the underlying distribution. (b) Within each segment, the sequences are generated with a simple Hidden Markov Model (HMM), with deterministic transition probability and emission probability.
  • Figure 3: The average MSE and Correlation between generated and natural DNA distributions for each model and motif type across 15 species.Fast A&E outperforms Hyena and DiscDiff, generating the most realistic sequences with the lowest MSE and highest Correlation across four motif types. This pattern is consistent across all 15 species.
  • Figure 4: Motif distributions in macaque DNA compared across natural DNA, FAST A&E, DiscDiff, and Hyena. FAST A&E closely aligns with natural DNA, especially for the TATA and GC motifs.
  • Figure 5: Evaluation of Generated Promoters for gene regulation through Genome Integration
  • ...and 17 more figures

Theorems & Definitions (1)

  • Proposition 1