DNATokenizer: A GPU-First Byte-to-Identifier Tokenizer for High-Throughput DNA Language Models

TL;DR

DNATok is vocabulary-agnostic: it accelerates single-nucleotide, non-overlapping k-mer, and BPE tokenization, and integrates as a drop-in systems layer beneath genomic foundation models, and achieves 84-95x higher encoding throughput than optimization baselines and up to 1.9x higher H2D throughput.

Abstract

Tokenization sits at the boundary between high-throughput genomic input and GPU compute, posing challenges in both algorithm design and system throughput. Overlapping k-mer tokenization can introduce information leakage under masked language modeling (MLM) and may degrade downstream accuracy. Single-nucleotide tokenization avoids leakage and preserves per-base fidelity, but it greatly increases sequence length for attention-based architectures. Non-overlapping k-mers and byte-pair encoding (BPE) provide compression and avoid leakage, at the cost of boundary sensitivity or reduced interpretability. Empirically, the choice of tokenization interacts strongly with model architecture and task requirements. At the system level, however, standard string tokenizers and host-bound vocabulary lookups dominate wall-clock time once inputs reach billions of bases, regardless of the tokenization algorithm. We present DNATok, a high-performance, GPU-first tokenization system that replaces general-purpose string processing with byte lookup table (LUT)-based identifier streaming and an overlapped host-to-device (H2D)/compute pipeline using pinned memory and architectural parallelism. DNATok is vocabulary-agnostic: it accelerates single-nucleotide, non-overlapping k-mer, and BPE tokenization, and integrates as a drop-in systems layer beneath genomic foundation models. DNATok achieves 84-95x higher encoding throughput than optimized Hugging Face baselines and up to 1.9x higher H2D throughput. End-to-end streaming reaches 1.27-1.84e8 tokens/s depending on configuration, effectively removing tokenization as a bottleneck for production-scale training and inference.

Figures (13)

• Figure 1: Encode throughput (tokens per second) by method for the standard configuration (batch size $B=4096$, sequence length $T=512$). DNATok variants outperform the Hugging Face tokenizer baseline by $90$--$95\times$, with staging i64 and encode i64 tied and staging i32 slightly lower; see Table \ref{['tab:benchmark_results']} for exact values.
• Figure 2: Host-to-Device throughput (tokens per second) by method for the standard configuration. Int32 staging delivers the highest H2D throughput by reducing transfer volume; staging i64 is slightly above the tokenizer baseline, while encode i64 is lower. Widening to 64-bit is performed on device.
• Figure 3: End-to-end throughput comparing baseline and pipelined streaming. In the standard configuration, pipelining slightly underperforms baseline streaming; see Table \ref{['tab:benchmark_results']} for measured values.
• Figure 4: Encode throughput versus batch size sweep. DNATok throughput rises with batch size and then plateaus, while tokenizer baselines remain relatively flat; trends align with the benchmark summary.
• Figure 5: End-to-end throughput versus batch size. Pipelined streaming shows mixed behavior: it underperforms at the smallest batches, exceeds baseline for several mid-to-large batches, and converges near parity at the largest batches.