When repeats drive the vocabulary: a Byte-Pair Encoding analysis of T2T primate genomes

TL;DR

This paper assesses the feasibility of using Byte-Pair Encoding to tokenize whole-genome sequences across nine telomere-to-telomere primate assemblies with a fixed 512,000-token vocabulary using a custom dnaBPE tokenizer. It shows that BPE captures abundant short repetitive motifs but yields a tiny core set of tokens shared across all genomes (only 0.6%), while the majority of tokens are species-specific, highlighting a strong bias toward high-copy repeats. Phylogenetic analyses based on token overlaps fail to reproduce known primate relationships, suggesting repeats dominate token patterns more than evolutionary signals. The authors propose hybrid tokenization strategies, including repeat masking and staged vocabularies, to mitigate these biases and enable more robust cross-genome modeling, and they provide open-source tools and data for reproducibility and further study.

Abstract

The emergence of telomere-to-telomere (T2T) genome assemblies has opened new avenues for comparative genomics, yet effective tokenization strategies for genomic sequences remain underexplored. In this pilot study, we apply Byte Pair Encoding (BPE) to nine T2T primate genomes including three human assemblies by training independent BPE tokenizers with a fixed vocabulary of 512,000 tokens using our custom tool, dnaBPE. Our analysis reveals that only 11,569 tokens are shared across all assemblies, while nearly 991,854 tokens are unique to a single genome, indicating a rapid decline in shared vocabulary with increasing assembly comparisons. Moreover, phylogenetic trees derived from token overlap failed to recapitulate established primate relationships, a discrepancy attributed to the disproportionate influence of species-specific high-copy repetitive elements. These findings underscore the dual nature of BPE tokenization: while it effectively compresses repetitive sequences, its sensitivity to high-copy elements limits its utility as a universal tool for comparative genomics. We discuss potential hybrid strategies and repeat-masking approaches to refine genomic tokenization, emphasizing the need for domain-specific adaptations in the development of large-scale genomic language models. The dnaBPE tool used in this study is open-source and available at https://github.com/aglabx/dnaBPE.

Figures (3)

• Figure 1: Analysis of token length distributions in the core set of 11,569 tokens shared across nine primate genomes. Left: Frequency histogram (log scale) showing the distribution of token lengths, with a pronounced peak at 8-12 bp and declining frequency of longer sequences. Right: Comparison of observed tokens (blue bars) versus theoretical maximum possible sequences (gray bars) for each length, demonstrating complete representation of 1-bp tokens (4/4), substantial coverage of 2-bp tokens (7/16), and rapidly decreasing coverage for longer sequences (e.g., only 0.34% of possible 10-bp sequences). This pattern suggests that BPE tokenization effectively captures short, conserved sequence motifs while longer tokens become increasingly species-specific.
• Figure 2: Comparative analysis of BPE token distributions across nine primate genomes. Left: Hierarchical clustering dendrogram based on token similarity reveals unexpected groupings that do not align with known primate phylogeny, with human genomes (HG, CN, CL) scattered across different clusters rather than forming a monophyletic group. Right: Heatmap of pairwise Jaccard distances between datasets shows high similarity among human assemblies (0.125-0.234) but unexpectedly high distances between evolutionarily close species, suggesting that BPE tokenization is strongly influenced by species-specific repetitive elements rather than evolutionary relationships. Color scale ranges from dark red (low distance, high similarity) to pale yellow (high distance, low similarity), highlighting the disconnect between token-based clustering and established primate phylogeny.
• Figure 3: Distribution of token sharing across nine primate genome tokenizers (log scale). The graph shows a dramatic decline in the number of shared tokens as the number of tokenizers increases, from nearly 1 million tokens unique to single genomes to only 11,569 tokens shared across all nine assemblies. The notable drop at four tokenizers reflects the underlying genomic distances: while human assemblies form a tight cluster (Jaccard distances 0.125-0.234), adding any fourth genome introduces substantial vocabulary divergence due to large evolutionary distances (Jaccard distances more than 0.9) between primate species. This pattern demonstrates how species-specific repetitive elements dominate BPE tokenization, challenging the development of a universal genomic tokenizer.