Text Indexing for Long Patterns using Locally Consistent Anchors

TL;DR

This work introduces a text index built on locally consistent anchors (lc-anchors), focusing on bidirectional anchors (bd-anchors) and their randomized variants to enable efficient long-pattern matching when a lower bound on pattern length $\ll$ is known. It combines a sampling scheme (A_{ll}(S) or A^{ran}_{ll}(S)) with compacted tries and a 2D range-reporting structure, achieving strong average-case guarantees with near-linear construction time and space proportional to the anchor set size $|_{ll}(S)| = O(n/\u001ell)$. The authors further improve construction via linear-time, average-case algorithms using (w,k)-minimizers and longest common extension queries, and present a small-space, near-optimal construction framework. They also derive a worst-case guarantees variant based on $ au$-partitioning sets, offering near-optimal queries with $O(n/\u001ell)$ space, albeit with randomized correctness. Extensive experiments on real datasets show randomized reduced bd-anchors outperform deterministic versions and classic indexes (suffix arrays, FM-index) for long patterns in both space and query time, with notable gains on the HUMAN genome.

Abstract

In many real-world database systems, a large fraction of the data is represented by strings: sequences of letters over some alphabet. This is because strings can easily encode data arising from different sources. It is often crucial to represent such string datasets in a compact form but also to simultaneously enable fast pattern matching queries. This is the classic text indexing problem. The four absolute measures anyone should pay attention to when designing or implementing a text index are: (i) index space; (ii) query time; (iii) construction space; and (iv) construction time. Unfortunately, however, most (if not all) widely-used indexes (e.g., suffix tree, suffix array, or their compressed counterparts) are not optimized for all four measures simultaneously, as it is difficult to have the best of all four worlds. Here, we take an important step in this direction by showing that text indexing with sampling based on locally consistent anchors (lc-anchors) offers remarkably good performance in all four measures, when we have at hand a lower bound $\ell$ on the length of the queried patterns -- which is arguably a quite reasonable assumption in practical applications. Our index offers average-case guarantees. In our experiments using real benchmark datasets, we show that it compares favorably based on the four measures to all classic indexes: (compressed) suffix tree; (compressed) suffix array; and the FM-index. Notably, we also present a counterpart of our index with worst-case guarantees based on the lc-anchors notion of partitioning sets. To the best of our knowledge, this is the first index achieving the best of all worlds in the regime where we have at hand a lower bound $\ell$ on the length of the queried patterns.

Figures (12)

• Figure 1: Let $S=\texttt{aacaaacgcta}$, $\ell=5$, and $r=1$. We have that $\mathcal{A}_{5,1}(S)=\{4,5,6,7\}$, and thus the indexed suffixes and reversed suffixes of $S$ are as shown in (a). Assume that we have a query pattern $P=\texttt{acaaa}$. We first find the reduced bd-anchor of $P[1\mathinner{.\,.} \ell]$ for $\ell=5$ and $r=1$, which is $j=3$: the minimal lexicographic rotation is aaaac. We then split $P$ in $\overleftarrow{P[1\mathinner{.\,.} 3]}=\texttt{aca}$ and $P[3\mathinner{.\,.} |P|]=\texttt{aaa}$, and search $\overleftarrow{P[1\mathinner{.\,.} 3]}=\texttt{aca}$ in $\mathcal{T}^L_{\ell,r}$ and $P[3\mathinner{.\,.} |P|]=\texttt{aaa}$ in $\mathcal{T}^R_{\ell,r}$. The search induces a rectangle, which encloses all the occurrences of $P$ in $S$, as shown in (b). Indeed, $P=S[2\mathinner{.\,.} 6]=\texttt{acaaa}$: the fragment $S[2 \mathinner{.\,.} 6]$ is anchored at position $4$.
• Figure 2: Illustration of Lemma \ref{['lem:LCP']}.
• Figure 3: Illustration of three applications of Lemma \ref{['lem:LCP']} on $F$. We mark the elements of $A$ in red circles.
• Figure 4: Illustration of the decomposition of $S$. We mark the $(w,k)$-minimizers in red circles.
• Figure 6: Number of bd-anchors output by rBDA-compute and rrBDA-compute for varying $\ell$ (log-scale).

Theorems & Definitions (33)

• Example 1: Minimizers
• Lemma 1: DBLP:journals/bioinformatics/ZhengKM20
• Lemma 2: DBLP:conf/esa/LoukidesP21
• Definition 1: Bidirectional anchor
• Example 2: Bd-anchors
• Definition 2: Reduced bidirectional anchor
• Example 3: Reduced bd-anchors
• Lemma 3: DBLP:conf/esa/LoukidesP21TKDE2023
• Lemma 4: DBLP:conf/esa/LoukidesP21TKDE2023
• Theorem 1: DBLP:conf/esa/LoukidesP21TKDE2023