Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tight Bounds and Phase Transitions for Incremental and Dynamic Retrieval

William Kuszmaul, Aaron Putterman, Tingqiang Xu, Hangrui Zhou, Renfei Zhou

TL;DR

This paper establishes optimal bounds for the final two settings (incremental and dynamic) in the case of a polynomial universe in the case of a polynomial universe.

Abstract

Retrieval data structures are data structures that answer key-value queries without paying the space overhead of explicitly storing keys. The problem can be formulated in four settings (static, value-dynamic, incremental, or dynamic), each of which offers different levels of dynamism to the user. In this paper, we establish optimal bounds for the final two settings (incremental and dynamic) in the case of a polynomial universe. Our results complete a line of work that has spanned more than two decades, and also come with a surprise: the incremental setting, which has long been viewed as essentially equivalent to the dynamic one, actually has a phase transition, in which, as the value size $v$ approaches $\log n$, the optimal space redundancy actually begins to shrink, going from roughly $n \log \log n$ (which has long been thought to be optimal) all the way down to $Θ(n)$ (which is the optimal bound even for the seemingly much-easier value-dynamic setting).

Tight Bounds and Phase Transitions for Incremental and Dynamic Retrieval

TL;DR

This paper establishes optimal bounds for the final two settings (incremental and dynamic) in the case of a polynomial universe in the case of a polynomial universe.

Abstract

Retrieval data structures are data structures that answer key-value queries without paying the space overhead of explicitly storing keys. The problem can be formulated in four settings (static, value-dynamic, incremental, or dynamic), each of which offers different levels of dynamism to the user. In this paper, we establish optimal bounds for the final two settings (incremental and dynamic) in the case of a polynomial universe. Our results complete a line of work that has spanned more than two decades, and also come with a surprise: the incremental setting, which has long been viewed as essentially equivalent to the dynamic one, actually has a phase transition, in which, as the value size approaches , the optimal space redundancy actually begins to shrink, going from roughly (which has long been thought to be optimal) all the way down to (which is the optimal bound even for the seemingly much-easier value-dynamic setting).

Paper Structure

This paper contains 21 sections, 12 theorems, 69 equations, 2 algorithms.

Table of Contents

  1. Introduction
  2. Our results.
  3. Upper Bounds for Incremental Retrieval
  4. Technical overview.
  5. Time-Inefficient Construction
  6. Improved Efficiency via Bucketing
  7. Constructing hash functions.
  8. Efficient universe reductions.
  9. Collision rates.
  10. Putting pieces together.
  11. Improved Efficiency via Linear Probing and Tiny Pointers
  12. Memory usage.
  13. Lower Bound for Incremental Retrieval Data Structures
  14. Lower Bound for Dynamic Retrieval Data Structures
  15. Lower Bound Set-up
  16. ...and 6 more sections

Key Result

Theorem 1.1

For universe size $|\mathcal{U}| = \mathop{\mathrm{poly}}\nolimits(n)$, there is an incremental retrieval data structure that supports all operations using an expected size of $S \leq nv + O(n) + O \mleft ( n \log \mleft ( \frac{\log(n)}{v} \mright ) \mright )$ bits.Strictly speaking, the space boun

Theorems & Definitions (73)

  • Theorem 1.1
  • Theorem 1.2
  • Theorem 1.3
  • Theorem 2.1
  • Remark 2.1
  • Definition 2.2
  • Claim 2.3
  • proof
  • Claim 2.4
  • proof
  • ...and 63 more