Intermediate N-Gramming: Deterministic and Fast N-Grams For Large N and Large Datasets

TL;DR

The paper tackles efficient, deterministic recovery of the top-$k$ $n$-grams from massive text and byte-sequence datasets. It introduces Intergrams, a cache-aware, multi-pass algorithm that builds candidate $n$-grams from top prefixes (e.g., $3$-grams) and progressively counts higher-order $n$-grams using only prefixes that survive prior passes, aided by a trie for fast membership checks. The approach is supported by theory under Zipfian distributions to bound recall and is validated by experiments showing substantial speedups (up to ~30x) over the prior Hash-Gramming method while maintaining high accuracy. The work demonstrates how hardware-aware design and prefix-based pruning can dramatically improve practical throughput for large-scale $n$-gram mining, with code available for reproduction.

Abstract

The number of n-gram features grows exponentially in n, making it computationally demanding to compute the most frequent n-grams even for n as small as 3. Motivated by our production machine learning system built on n-gram features, we ask: is it possible to accurately, deterministically, and quickly recover the top-k most frequent n-grams? We devise a multi-pass algorithm called Intergrams that constructs candidate n-grams from the preceding (n - 1)-grams. By designing this algorithm with hardware in mind, our approach yields more than an order of magnitude speedup (up to 33x!) over the next known fastest algorithm, even when similar optimizations are applied to the other algorithm. Using the empirical power-law distribution over n-grams, we also provide theory to inform the efficacy of our multi-pass approach. Our code is available at https://github.com/rcurtin/Intergrams.

Figures (5)

• Figure 1: A typical machine learning pipeline using $n$-grams as features. To run this pipeline, the $k$ most common $n$-grams from the input data must be known. Often, on large datasets, the process of computing the top-$k$$n$-grams is far more expensive than the modeling step!
• Figure 2: Memory bandwidth benchmarks. We vary the size of the array (x axis), and in each trial we increment all elements of the array in sequential (blue) or random (red) order. For sequential access patterns, the hardware prefetcher can keep bandwidth high, but the 2048-element TLB necessarily starts to miss at 8 MB (we use 4 KB page sizes). For random access patterns, the hardware prefetcher cannot predict the memory (or page) that is needed next, and throughput craters as the array size exceeds the size of each level in the cache hierarchy.
• Figure 3: Percentage of top-$k$ 4-grams that have 3-byte prefixes in the top-$zk$ 3-grams on the EMBER dataset, as a function of $z$. 100% means that all 4-grams have 3-byte prefixes in the top-$zk$ 3-grams; in this case, we can filter the set of possible 4-grams significantly, with no loss of accuracy!
• Figure 4: An example trie built on length-3 byte sequence data. Determining whether a prefix $x$ is in $P$ can be done by iterating from the root of the trie to a leaf, considering each element of the prefix in succession. In this trie, we can determine that the byte sequence 0x013300 is not in $P$ at the first iteration. Other sequences, like 0x000aaf and 0xff00ff require more iterations; sequences like 0x000000 are in $P$ and will reach a leaf during iteration of the trie.
• Figure 5: Runtime results for a sweep of $k$ on the EMBER dataset. Y-axis is logarithmic.

Theorems & Definitions (13)

• Lemma 1
• Theorem 1
• Lemma 2
• Theorem 2
• Lemma 2
• proof
• Theorem 2
• Theorem 2
• proof
• Lemma 2