Memory-Efficient Sequential Pattern Mining with Hybrid Tries

TL;DR

This work tackles the memory bottleneck in sequential pattern mining (SPM) by introducing memory-efficient data structures and algorithms. The authors first develop a binary trie (BT) representation with an ancestor label to compactly store overlapping prefixes, then extend to a hybrid trie-vector model (HT) that transitions from a BT for prefixes to a vector postfix, controlled by a data-dependent transition length $j^*$. The integrated HTMiner algorithm combines BTMiner for prefixes with prefix-projection for postfixes, achieving superior memory efficiency and often faster runtimes compared to state-of-the-art vector and trie-based methods; HTMiner is able to mine large data sets within 256GB of memory, potentially saving up to 1.7TB relative to vector-based baselines. These contributions enable scalable SPM on large-scale data sets and offer a practical path for memory-constrained environments in domains like fraud detection and market analysis. The results demonstrate substantial memory reductions across diverse datasets while maintaining competitive or improved computational performance, highlighting the practical impact of the hybrid trie-vector approach for large-scale SPM.

Abstract

This paper develops a memory-efficient approach for Sequential Pattern Mining (SPM), a fundamental topic in knowledge discovery that faces a well-known memory bottleneck for large data sets. Our methodology involves a novel hybrid trie data structure that exploits recurring patterns to compactly store the data set in memory; and a corresponding mining algorithm designed to effectively extract patterns from this compact representation. Numerical results on small to medium-sized real-life test instances show an average improvement of 85% in memory consumption and 49% in computation time compared to the state of the art. For large data sets, our algorithm stands out as the only capable SPM approach within 256GB of system memory, potentially saving 1.7TB in memory consumption.

Figures (6)

• Figure 1: General trie model $\mathcal{T}$ of the data set in Table \ref{['tab:vector']}. Although the general trie structure correctly models all sequences of the data set and their frequencies, its labels are insufficient for SPM.
• Figure 2: Trie model of TreeMiner for the data set of Table \ref{['tab:vector']}. The trie is updated with itemset labels $\mathbb{I}$, next-links displayed as a matrix adjacent to each node, and the parent-info bitset displayed as a vector under each node. TreeMiner mines the trie by traversing it using next-links and determining valid extensions using parent-info and itemset labels.
• Figure 3: The $\mathcal{BT}$ model of the data set in Table \ref{['tab:vector']}. Sibling nodes are connected by dashed arcs. The trie is updated with itemset labels $\mathbb{I}_n$ and ancestor labels $a_n$, which are sufficient for SPM.
• Figure 4: An $\mathcal{HT}$ model of the data set in Table \ref{['tab:vector']}. The model transitions from a trie to a vector representation at layer $\mathcal{N}_4$. Transitioning nodes $n_4, n_{10}$ are associated to an ancestor vector that tracks the ancestors for all events $e \in \mathbb{E}: e \in \mathbf{S}({\bf{r}}, \dots, n_j), j \in \{4,10\}$.
• Figure 5: Peak memory consumption. Algorithms that exceed system memory are shown in gray.

Theorems & Definitions (25)

• Example 1
• Example 2
• Lemma 3
• proof
• Example 4
• Lemma 5
• proof
• Definition 6
• Example 7
• Proposition 8