Static Retrieval Revisited: To Optimality and Beyond

TL;DR

This work characterizes the fundamental time–space limits of static retrieval, proving a tight cell-probe lower bound that links query time, word size, and redundancy, and showing a strong trade-off even for large value sizes $v$. It then introduces augmented retrieval, coupling a retrieval data structure with an auxiliary array, to bypass the lower bound and achieve fast queries with near-optimal space $nv + \mathrm{Space}(D_2) + n^{0.67}$, under suitable conditions. A parallel development shows the same time/space curve applies to static filters via a standard reduction, establishing near-optimal space for membership with worst-case probes. The paper then delivers constructive upper bounds for augmented retrieval using a sparse, random-matrix framework with coupon and permutation entries, proves the key full-rank property with high probability, and extends the results to remove randomness assumptions and to handle small value universes. The open questions highlight gaps for non-power-of-two domains, tighter redundancy bounds in the augmented setting, and broader applicability of the augmented-data primitives.

Abstract

In the static retrieval problem, a data structure must answer retrieval queries mapping a set of $n$ keys in a universe $[U]$ to $v$-bit values. Information-theoretically, retrieval data structures can use as little as $nv$ bits of space. For small value sizes $v$, it is possible to achieve $O(1)$ query time while using space $nv + o(n)$ bits -- whether or not such a result is possible for larger values of $v$ (e.g., $v = Θ(\log n)$) has remained open. In this paper, we obtain a tight lower bound (as well as matching upper bounds) for the static retrieval problem. In the case where values are large, we show that there is actually a significant tension between time and space. It is not possible, for example, to get $O(1)$ query time using $nv + o(n)$ bits of space, when $v = Θ(\log n)$ (and assuming the word RAM model with $O(\log n)$-bit words). At first glance, our lower bound would seem to render retrieval unusable in many settings that aim to achieve very low redundancy. However, our second result offers a way around this: We show that, whenever a retrieval data structure $D_1$ is stored along with another data structure $D_2$ (whose size is similar to or larger than the size of $D_1$), it is possible to implement the combined data structure $D_1 \cup D_2$ so that queries to $D_1$ take $O(1)$ time, operations on $D_2$ take the same asymptotic time as if $D_2$ were stored on its own, and the total space is $nv + \mathrm{Space}(D_2) + n^{0.67}$ bits.

Theorems & Definitions (38)

• Theorem 1.1
• Theorem 1.2
• Theorem 1.3: Later Restated as \ref{['thm:non_augmented_lb_filter']}
• Theorem 2.1
• proof
• Lemma 2.3: vondrak2010note, rephrased
• Claim 2.4
• proof
• Theorem 2.5
• Theorem 2.6