Indexing Strings with Utilities

TL;DR

This work introduces Useful String Indexing (USI), a framework to preprocess a weighted string so that the global utility $U(P)$ of any pattern $P$ can be queried efficiently. The authors combine a top-$K$ frequent-substring index with a classic text index, enabling fast queries while keeping space near linear in $n$ and $K$. They present exact and space-efficient approximate methods for identifying top-$K$ substrings, with detailed construction and query-time analyses, and demonstrate substantial speedups (up to 15x) over baselines in large-scale experiments. The results show USI’s potential to accelerate pattern evaluation in domains such as bioinformatics, advertising, and web analytics, where patterns are short but occur billions of times with per-letter utilities. The work also discusses limitations of adapting existing frequent-item mining algorithms to substrings and offers practical guidance for parameter choices and future dynamic extensions.

Abstract

Applications in domains ranging from bioinformatics to advertising feature strings that come with numerical scores (utilities). The utilities quantify the importance, interest, profit, or risk of the letters occurring at every position of a string. Motivated by the ever-increasing rate of generating such data, as well as by their importance in several domains, we introduce Useful String Indexing (USI), a natural generalization of the classic String Indexing problem. Given a string $S$ (the text) of length $n$, USI asks for preprocessing $S$ into a compact data structure supporting the following queries efficiently: given a shorter string $P$ (the pattern), return the global utility $U(P)$ of $P$ in $S$, where $U$ is a function that maps any string $P$ to a utility score based on the utilities of the letters of every occurrence of $P$ in $S$. Our work also makes the following contributions: (1) We propose a novel and efficient data structure for USI based on finding the top-$K$ frequent substrings of $S$. (2) We propose a linear-space data structure that can be used to mine the top-$K$ frequent substrings of $S$ or to tune the parameters of the USI data structure. (3) We propose a novel space-efficient algorithm for estimating the set of the top-$K$ frequent substrings of $S$, thus improving the construction space of the data structure for USI. (4) We show that popular space-efficient top-$K$ frequent item mining strategies employed by state-of-the-art algorithms do not smoothly translate from items to substrings. (5) Using billion-letter datasets, we experimentally demonstrate that: (i) our top-$K$ frequent substring mining algorithms are accurate and scalable, unlike two state-of-the-art methods; and (ii) our USI data structures are up to $15$ times faster in querying than $4$ nontrivial baselines while occupying the same space with them.

Figures (6)

• Figure 1: Illustration of real strings with associated utilities.
• Figure 2: Overview of our approach for USI.
• Figure 3: Accuracy vs (a-e) $K$, (f-i) $n$, and (j) $s$ ($s$ affects only $\textsf{AT}\xspace$). We omit SH when it did not terminate within $5$ days.
• Figure 4: Accuracy vs (a-c) $s$. NDCG (d) for all datasets and (e) vs $s$. We omit SH when it did not terminate within $5$ days.
• Figure 5: Space for ET, AT, TT, and SH vs (a, b) $n$ and (c, d) $s$ (recall that parameter $s$ affects only $\textsf{AT}\xspace$). Runtime for ET, AT, TT, and SH (e, f) $K$, (g, h) $n$, and (i, j) $s$. We omit SH when it did not terminate within $5$ days.

Theorems & Definitions (5)

• Example 1
• Example 2
• Theorem 1
• Theorem 2
• Theorem 3