PtrHash: Minimal Perfect Hashing at RAM Throughput

TL;DR

PtrHash addresses the problem of designing a minimal perfect hash function that prioritizes query throughput and construction speed while keeping space under 3 bits/key. It advances prior work by using fixed-width 8-bit pilots, a fixed number of buckets per part, a single global remap encoded with CacheLineEF, and streaming-prefetching to exploit memory bandwidth. The approach delivers around 2.4 bits/key of space with fast construction and substantially higher query throughput (up to 3.3× faster in streaming scenarios) than alternatives, approaching the limits set by main memory bandwidth for large datasets ($n$ on the order of $10^9$). The work has practical impact for large-scale indexing in bioinformatics and data-intensive domains where rapid lookups and streaming queries are valuable, enabling near-RAM bandwidth throughput on modern hardware.

Abstract

Given a set $K$ of $n$ keys, a minimal perfect hash function (MPHF) is a collision-free bijective map $\mathsf{H_{mphf}}$ from $K$ to $\{0, \dots, n-1\}$. This work presents a (minimal) perfect hash function that first prioritizes query throughput, while also allowing efficient construction for $10^9$ or more elements using 2.4 bits of memory per key. Both PTHash and PHOBIC first map all $n$ keys to $n/λ< n$ buckets. Then, each bucket stores a pilot that controls the final hash value of the keys mapping to it. PtrHash builds on this by using 1) fixed-width (uncompressed) 8-bit pilots, 2) a construction algorithm similar to cuckoo-hashing to find suitable pilot values. Further, it 3) uses the same number of buckets and slots for each part, with 4) a single remap table to map intermediate positions $\geq n$ to $<n$, 5) encoded using per-cacheline Elias-Fano coding. Lastly, 6) PtrHash support streaming queries, where we use prefetching to answer a stream of multiple queries more efficiently than one-by-one processing. With default parameters, PtrHash takes 2.0 bits per key. On 300 million string keys, PtrHash is as fast or faster to build than other MPHFs, and at least $2.1\times$ faster to query. When streaming multiple queries, this improves to $3.3\times$ speedup over the fastest alternative, while also being significantly faster to construct. When using $10^9$ integer keys instead, query times are as low as 12 ns/key when iterating in a for loop, or even down to 8 ns/key when using the streaming approach, just short of the 7.4 ns inverse throughput of random memory accesses.

Figures (7)

• Figure 1: Overview of PtrHash on $n=23$ keys. The keys are hashed into $[H] = [2^{64}]$ and this range is split into $P=2$ parts and $B=5$ buckets per part. The key highlighted in yellow has a the 9'th smallest hash, and ends up in bucket 4 (starting at index 0). The corresponding pilot$p_4$ hashes the key to slot 6. The array of pilots (grey background) is the main component of the PtrHash data structure, and ensures that all keys hash to different slots. The blue key has a hash in the second part (upper half) of hashes, in bucket 6. It gets hashed to slot 25, which is larger than the number of keys $n=23$. Thus, it is remapped (along with the other red cells) into an empty slot $<n$ via a (compressed) list of free slots, which is the second main component of the data structure.
• Figure 2: The left shows various bucket assignment functions $\gamma$, such as the piecewise linear function (skewed) used by FCH and PTHash, and the optimal function introduced by PHOBIC. Flatter slopes at $x=0$ create larger buckets, while steeper slopes at $x=1$ create more small buckets, as shown on the right, as the distribution of expected bucket sizes given by $(\gamma^{-1})'$ when the expected bucket size is $\lambda=4$.
• Figure 3: Overview of the CacheLineEF data structure.
• Figure 4: Bucket size distribution (red) and average number of evictions (black) per additionally placed bucket during construction of the pilot table, for different bucket assignment functions. Parameters are $n=10^9$ keys, $S=2^{18}$ slots per part, and $\alpha=0.99$, and the red shaded load factor ranges from 0 to $\alpha$. In the first five plots $\lambda=3.5$ so that the pilots take 2.29 bits/key. For $\lambda=4.0$ (bottom-right), the linear, skewed, and optimal bucket assignment functions cause endless evictions, and construction fails. The cubic function does work, resulting in 2.0 bits/key for the pilots.
• Figure 5: This plot shows the construction time (blue and red, left axis) and data structure size (black, green, and yellow, right axis) as a function of $\lambda$ for $n=10^9$ keys. Parallel construction time on 6 threads is shown for both the linear and cubic $\gamma$, and for various values of $\alpha$ (thickness). The curves stop because construction times out when $\lambda$ is too large. For each $\lambda$, the black line shows the space taken by the array of pilots. For larger $\lambda$ there are fewer buckets, and hence the pilots take less space. The total size including the remap table is shown in green (plain vector) and yellow (CacheLineEF) for various $\alpha$. The blue (fast), black (default), and red (compact) dots highlight the chosen parameter configurations.