PHOBIC: Perfect Hashing with Optimized Bucket Sizes and Interleaved Coding

TL;DR

PHOBIC advances minimal perfect hash function construction by optimizing bucket size distributions via an optimal bucket assignment function and by encoding seeds with an interleaved scheme across partitions. It introduces a perturbation to cap bucket sizes for practicality and provides a GPU-accelerated construction path, achieving about 0.17 bits/key space improvement over PTHash and up to 62x faster GPU-based construction. The approach maintains fast query performance while reducing space and build time, bringing MPHFs closer to the information-theoretic lower bound in practice. This work has practical impact for large static data structures in bioinformatics and databases, enabling faster and more space-efficient indexing at scale.

Abstract

A minimal perfect hash function (MPHF) maps a set of n keys to {1, ..., n} without collisions. Such functions find widespread application e.g. in bioinformatics and databases. In this paper we revisit PTHash - a construction technique particularly designed for fast queries. PTHash distributes the input keys into small buckets and, for each bucket, it searches for a hash function seed that places its keys in the output domain without collisions. The collection of all seeds is then stored in a compressed way. Since the first buckets are easier to place, buckets are considered in non-increasing order of size. Additionally, PTHash heuristically produces an imbalanced distribution of bucket sizes by distributing 60% of the keys into 30% of the buckets. Our main contribution is to characterize, up to lower order terms, an optimal distribution of expected bucket sizes. We arrive at a simple, closed form solution which improves construction throughput for space efficient configurations in practice. Our second contribution is a novel encoding scheme for the seeds. We split the keys into partitions. Within each partition, we run the bucket distribution and search step. We then store the seeds in an interleaved way by consecutively placing the seeds for the i-th buckets from all partitions. The seeds for the i-th bucket of each partition follow the same statistical distribution. This allows us to tune a compressor for each bucket. Hence, we call our technique PHOBIC - Perfect Hashing with Optimized Bucket sizes and Interleaved Coding. Compared to PTHash, PHOBIC is 0.17 bits/key more space efficient for same query time and construction throughput. We also contribute a GPU implementation to further accelerate MPHF construction. For a configuration with fast queries, PHOBIC-GPU can construct a perfect hash function at 2.17 bits/key in 28 ns per key, which can be queried in 37 ns on the CPU.

Figures (10)

• Figure 1: Comparison of bucket assignment functions $\gamma(x)$ of related work and PHOBIC ($\gamma=\beta_*$).
• Figure 2: Interleaved coding. Encoder $i$ stores the seed of bucket $i$ from all partitions.
• Figure 3: Each box represents one seed tested by one thread. Left: Synchronized nested loop. Right: \ref{['alg:simt']} where we continue with the first key and the next seed after a collision.
• Figure 4: Comparing original PTHash (EF, $\alpha=0.99$) to fine-grained partitioning with additive displacement hash (mono-R). We then add interleaved coding (IC-R) and optimized bucketing (OB) individually. Putting it all together we arrive at PHOBIC. All single-threaded. There are small differences in query time.
• Figure 5: Query time and space consumption of Elias-Fano, Dictionary, Compact, and Rice encoders. Points prefixed "mono" place all seeds into a single encoder and those prefixed "IC" use interleaved coding. The curve shows different mixtures (see \ref{['s:partitioning']}) of Compact and Rice encoders in the interleaved coding.

Theorems & Definitions (14)

• Proposition 1
• Proposition 2
• Theorem 4
• Theorem 5
• Definition 7
• Lemma 8: Deadline Lemma
• Definition 9
• Lemma 12
• Corollary 13
• Claim 14