Succinct Dynamic Rank/Select: Bypassing the Tree-Structure Bottleneck

TL;DR

This work introduces a dynamic Fully Indexable Dictionary that circumvents the traditional tree-structure bottleneck by leveraging a history-independent, space-efficient tabulation-weighted treap. The treap, augmented with a carefully designed weight function and a recursive information-model encoding, supports insert/delete/rank/select in amortized time close to optimal while achieving redundancy as low as $n/2^{(\log n)^{\Omega(1)}}$ bits for a universe $U = \operatorname{poly}(n)$. A transfer to the RAM model uses virtual memory, adapters, spillover representations, and entropy encoders to realize deterministic, space-efficient encodings with $O(\log \binom{U}{n} + w n / 2^{(\log n/\log w)^{1/3}})$ bits and polylogarithmic factors in time. The approach highlights history independence as a central design principle for space efficiency and outlines a robust framework for combining compressed data structures with probabilistic encodings. The results resolve a longstanding open question by showing a dynamic FID with sublinear redundancy and near-optimal time bounds, broadening the practical reach of succinct data structures in dynamic settings.

Abstract

We show how to construct a dynamic ordered dictionary, supporting insert/delete/rank/select on a set of $n$ elements from a universe of size $U$, that achieves the optimal amortized expected time complexity of $O(1 + \log n / \log \log U)$, while achieving a nearly optimal space consumption of $\log \binom{U}{n} + n / 2^{(\log n)^{Ω(1)}} + \text{polylog}\, U$ bits in the regime where $U = \text{poly}(n)$. This resolves an open question by Pibiri and Venturini as to whether a redundancy (a.k.a. space overhead) of $o(n)$ bits is possible, and is the first dynamic solution to bypass the so-called tree-structure bottleneck, in which the bits needed to encode some dynamic tree structure are themselves enough to force a redundancy of $\widetildeΩ(n)$ bits. Our main technical building block is a dynamic balanced binary search tree, which we call the compressed tabulation-weighted treap, that itself achieves a surprising time/space tradeoff. The tree supports $\text{polylog}\, n$-time operations and requires a static lookup table of size $\text{poly}(n) + \text{polylog}\, U$ -- but, in exchange for these, the tree is able to achieve a remarkable space guarantee. Its total space redundancy is $O(\log U)$ bits. In fact, if the tree is given $n$ and $U$ for free, then the redundancy further drops to $O(1)$ bits.

Figures (3)

• Figure 1: Encoding process in the large-universe case. In Step 1, we start with the spillover representations of the two subtrees, and combine them using an adapter. In Steps 2--4, we use three entropy encoders to add $r$, $p_{\textup{high-mid}}$, and $h(p)$ to the representation, respectively. In Step 5, we reduce the size of the spill and form the final output.
• Figure 2: Encoding process in the small-universe case. We start with the spillover representations of the two subtrees, and combine them using an adapter. Then, we use an entropy encoder to add $(\Delta p,r)$ to the representation. Finally, we reduce the size of the spill and form the final output.
• Figure 3: Encoding process of the entropy encoder. We first partition $m_{\textup{in}}$ into $m_{\textup{fix}}$ and $m_{\textup{rem}}$, and split $k_{\textup{in}}$ into $m_{\mathrm{spill}}$ and $k_{\textup{short}}$. We then encode $(\varphi, m_{\textup{rem}}, m_{\mathrm{spill}}, k_{\textup{short}})$ to $(m_{\textup{enc}}, k_{\textup{enc}})$ using \ref{['lem:succincter-lem5']} under the perturbed distribution $\widetilde{D}$. Finally, we output $(m_{\textup{out}}, k_{\textup{out}})$, where $m_{\textup{out}} = m_{\textup{fix}}$ and $k_{\textup{out}}$ is composed of $m_{\textup{enc}}$ and $k_{\textup{enc}}$.

Theorems & Definitions (41)

• Theorem 1.1
• Proposition 1.2: Summarized version of \ref{['lem:treap']}
• Proposition 2.0: Compressed Tabulation-Weighted Treaps
• Lemma 3.1
• proof
• Claim 4.1
• proof
• Claim 4.2
• proof
• Claim 4.3