Suffix sorting via matching statistics

TL;DR

This work tackles generalized suffix array construction for collections of highly similar strings by introducing a compressed matching statistics (CMS) framework anchored to a reference string $R$. An enhanced CMS (eCMS) with insert-heads enables fast, locality-friendly suffix comparisons, which are then organized through a multi-phase algorithm that buckets, sorts, and induces suffixes to build the GSA with worst-case time $O(N\log N)$, where $N=\sum_d |S_d|$. The authors implement a prototype, sacamats, and show it is competitive with or faster than state-of-the-art tools on highly repetitive data such as viral genomes and large chromosome datasets, while also providing a practical fast CMS computation heuristic. The CMS approach promises practical gains for pangenomic indexing and distributed or sparse suffix sorting, offering a new avenue for scalable genome-scale text indexing.

Abstract

We introduce a new algorithm for constructing the generalized suffix array of a collection of highly similar strings. As a first step, we construct a compressed representation of the matching statistics of the collection with respect to a reference string. We then use this data structure to distribute suffixes into a partial order, and subsequently to speed up suffix comparisons to complete the generalized suffix array. Our experimental evidence with a prototype implementation (a tool we call sacamats) shows that on string collections with highly similar strings we can construct the suffix array in time competitive with or faster than the fastest available methods. Along the way, we describe a heuristic for fast computation of the matching statistics of two strings, which may be of independent interest.

Figures (4)

• Figure 1: Example for the matching statistics and the data for the CMS and the eCMS. In the first two rows, we give $\textit{MS}$ of $S$ w.r.t. $R$, where $\textit{MS}[i]=(p_i,\ell_i)$. In row 3, we mark the heads (for the CMS). In rows 4, we give the position $q_i$, defined by $ip(i)$, i.e. $q_i=\textit{SA}_R[ip(i)]$, where $ip(i)$ is the insert-point of suffix $S[i..]$ in the suffix array of $R$. In row $5$, we mark the insert-heads (for the eCMS).
• Figure 2: Details of computation of the matching statistics from Figure \ref{['fig:example_eCMS']}. We highlight in blue the matching factors for the indices $i=9$ (matching factor ${\tt AT}$, mismatch character ${\tt T}$) and $11$ (matching factor ${\tt TGATGG}$, mismatch character ${\tt \$}$). The arrows represent the insert-points.
• Figure 3: Experiments on different subsets of copies of Human Chromosome 19.
• Figure 4: Experiments on different subsets of SARS-CoV2 genomes.

Theorems & Definitions (15)

• Definition 1: Compressed matching statistics
• Lemma 2
• Example 3
• Proposition 4
• Definition 5: Enhanced compressed matching statistics
• Example 6
• Lemma 8
• Definition 9: Right extension and left contraction
• Lemma 10
• Lemma 11