Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hecate: A Modular Genomic Compressor

Kamila Szewczyk, Sven Rahmann

Abstract

We present Hecate, a modular lossless genomic compression framework. It is designed around uncommon but practical source-coding choices. Unlike many single-method compressors, Hecate treats compression as a conditional coding problem over coupled FASTA/FASTQ streams (control, headers, nucleotides, case, quality, extras). It uses per-stream codecs under a shared indexed block container. Codecs include alphabet-aware packing with an explicit side channel for out-of-alphabet residues, an auxiliary-index Burrows-Wheeler pipeline with custom arithmetic coding, and a blockwise Markov mixture coder with explicit model-competition signaling. This architecture yields high throughput, exact random-access slicing, and referential mode through streamwise binary differencing. In a comprehensive benchmark suite, Hecate provides the best compression vs. speed trade-offs against state-of-the-art established tools (MFCompress, NAF, bzip3, AGC), with notably stronger behaviour on large genomes and high-similarity referential settings. For the same compression ratio, Hecate is 2 to 10 times faster. When given the same time budget as other algorithms, Hecate achieves up to 5% to 10% better compression.

Hecate: A Modular Genomic Compressor

Abstract

We present Hecate, a modular lossless genomic compression framework. It is designed around uncommon but practical source-coding choices. Unlike many single-method compressors, Hecate treats compression as a conditional coding problem over coupled FASTA/FASTQ streams (control, headers, nucleotides, case, quality, extras). It uses per-stream codecs under a shared indexed block container. Codecs include alphabet-aware packing with an explicit side channel for out-of-alphabet residues, an auxiliary-index Burrows-Wheeler pipeline with custom arithmetic coding, and a blockwise Markov mixture coder with explicit model-competition signaling. This architecture yields high throughput, exact random-access slicing, and referential mode through streamwise binary differencing. In a comprehensive benchmark suite, Hecate provides the best compression vs. speed trade-offs against state-of-the-art established tools (MFCompress, NAF, bzip3, AGC), with notably stronger behaviour on large genomes and high-similarity referential settings. For the same compression ratio, Hecate is 2 to 10 times faster. When given the same time budget as other algorithms, Hecate achieves up to 5% to 10% better compression.
Paper Structure (25 sections, 13 equations, 1 figure, 3 tables)

This paper contains 25 sections, 13 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Burrows-Wheeler Transform and LF Mapping
  4. Prediction by Partial Matching (PPM)
  5. Methods: Reference-Free Compression
  6. Container and Preprocessing
  7. hecate-bwt
  8. BWT construction and auxiliary-index inversion.
  9. Bitwise probability model.
  10. Counter dynamics.
  11. Parallel chunked coding.
  12. markov-mix
  13. Expert family.
  14. Per-context prediction.
  15. Expert selection.
  16. ...and 10 more sections

Figures (1)

  • Figure 1: Encode and decode times for GCA_000404065.3.fna (Pinus taeda, 22.5 Gb), GCF_000001405.40.fna (Homo sapiens (GRCh38.p14), 3.3 Gb), GCA_004837865.1.fna (Musa balbisiana, 499 Mb), GCF_009914755.1.fna (Homo sapiens (T2T-CHM13v2.0), 3.2 Gb) and GCA_021556685.1.fna (Rattus norvegicus, 2.9 Gb).