Competitive Data-Structure Dynamization

TL;DR

The paper introduces data-structure dynamization through two online set-cover problems that capture non-uniform input patterns common in practical LSM-based systems. By applying competitive analysis, it develops deterministic online algorithms—Adaptive-Binary for Min-Sum Dynamization with a $Θ( ext{log}^* m)$ competitive ratio and Greedy-Dual/B_k variants for $k$-Component Dynamization with a $Θ(k)$ ratio—that substantially improve over worst-case transforms. It also proves that optimal offline solutions are newest-first and lightest-first, enabling near-optimality for uniform inputs and enabling polynomial-time computation of optimal strategies in certain variants. The work bridges theory and practice by aligning the models with real-world systems like Bigtable/LevelDB, and discusses practical extensions, variations, and open problems, including energy-aware, external-memory considerations and stability under tombstones, splits, and Bloom filters. Overall, the results provide principled, worst-case guarantees while exploiting non-uniform workloads to reduce costs in dynamic storage and query-heavy environments.

Abstract

Data-structure dynamization is a general approach for making static data structures dynamic. It is used extensively in geometric settings and in the guise of so-called merge (or compaction) policies in big-data databases such as Google Bigtable and LevelDB (our focus). Previous theoretical work is based on worst-case analyses for uniform inputs -- insertions of one item at a time and constant read rate. In practice, merge policies must not only handle batch insertions and varying read/write ratios, they can take advantage of such non-uniformity to reduce cost on a per-input basis. To model this, we initiate the study of data-structure dynamization through the lens of competitive analysis, via two new online set-cover problems. For each, the input is a sequence of disjoint sets of weighted items. The sets are revealed one at a time. The algorithm must respond to each with a set cover that covers all items revealed so far. It obtains the cover incrementally from the previous cover by adding one or more sets and optionally removing existing sets. For each new set the algorithm incurs build cost equal to the weight of the items in the set. In the first problem the objective is to minimize total build cost plus total query cost, where the algorithm incurs a query cost at each time $t$ equal to the current cover size. In the second problem, the objective is to minimize the build cost while keeping the query cost from exceeding $k$ (a given parameter) at any time. We give deterministic online algorithms for both variants, with competitive ratios of $Θ(\log^* n)$ and $k$, respectively. The latter ratio is optimal for the second variant.

Figures (9)

• Figure 1: Steps 1--11 of the binary transform bentley_decomposable_1979bentley_decomposable_1980. Each cell i is a component holding $i$ items, where $i$ is a distinct power of two. In each step one item is inserted and held in the new (top, bolded) component.
• Figure 2: Steps 1--11 of the $2$-binomial transform bentley_decomposable_1980. At time $t$ the top and bottom components hold $\binom{i_1} 1$ and $\binom{i_2} 2$ items where $0\le i_1<i_2$ and $\binom{i_1} 1 + \binom{i_2} 2 = t$. For example at time t = 8, $i_1 = 2$ and $i_2 = 4$. If $i_1=0$ there is only one component, the bottom component.
• Figure 3: A $\Theta(\log^* m)$-competitive algorithm for Min-Sum Dynamization (Theorem \ref{['thm: min-sum']}).
• Figure 4: The capacities $\mu(t)$ as a function of $t$.
• Figure 5: The "merge tree" for an execution of the Adaptive-Binary algorithm (Figure \ref{['fig: min-sum alg']}). The input sequence starts with $m=132$inserts $I_1, I_2, \ldots, I_{132}$ --- one for each leaf, of weight equal to leaf's label. It continues with $2^{16} - 132$ empty inserts ($I_t=\emptyset$). At each time $t=2^9, 2^{10}, 2^{11}, \ldots, 2^{17}$ (during the empty inserts) the algorithm merges all components of weight $t$ to form a single new component, their parent. In this way, the algorithm builds a component for each node, with weight equal to the node's label. At time $t=2^{17}$ the final component is built---the root, of weight $2^{18}$, containing all items. The algorithm merges each item four times, so pays build cost $4\times 2^{18}$.

Theorems & Definitions (17)

• theorem \ref{thm: min-sum}: Section \ref{['sec: min-sum']}
• theorem \ref{thm: k-component lower bound}: Section \ref{['sec: k-component lower bound']}
• theorem \ref{thm: k-component decreasing}: Section \ref{['sec: k-component decreasing']}
• definition 1
• theorem \ref{thm: meta}: Section \ref{['sec: k-component LSM']}
• corollary \ref{cor: k-component LSM}: Section \ref{['sec: k-component LSM']}
• theorem \ref{thm: k-component general}: Section \ref{['sec: k-component general']}
• theorem \ref{thm: offline}: Section \ref{['sec: offline']}
• lemma 1
• definition 2: dominant