Memory Reallocation with Polylogarithmic Overhead

TL;DR

This work advances the Memory Reallocation problem by introducing a randomized resizable allocator that achieves polylogarithmic worst-case expected overhead in the load-factor deficit $\varepsilon$, significantly surpassing previous bounds. The approach leverages additive combinatorics, notably the sunflower lemma and Erdős–Sárközy equal-sum subsets, to bundle objects into a bounded number of sizes, enabling efficient substitutions with periodic rebuilds. A dual contribution establishes a logarithmic lower bound for expected overhead and a polynomial lower bound for expected squared overhead, proving that high-probability subpolynomial overhead is impossible in general. The combination of bundling and a structured rebuild strategy yields a near-optimal, polylogarithmic expected overhead while clarifying fundamental limits of tail guarantees. Overall, the paper demonstrates that sunflower-based grouping can powerfully inform dynamic memory allocation with reallocation, and it outlines open questions on time complexity and high-probability performance.

Abstract

The Memory Reallocation problem asks to dynamically maintain an assignment of given objects of various sizes to non-overlapping contiguous chunks of memory, while supporting updates (insertions/deletions) in an online fashion. The total size of live objects at any time is guaranteed to be at most a $1-ε$ fraction of the total memory. To handle an online update, the allocator may rearrange the objects in memory to make space, and the overhead for this update is defined as the total size of moved objects divided by the size of the object being inserted/deleted. Our main result is an allocator with worst-case expected overhead $\mathrm{polylog}(ε^{-1})$. This exponentially improves the previous worst-case expected overhead $\tilde O(ε^{-1/2})$ achieved by Farach-Colton, Kuszmaul, Sheffield, and Westover (2024), narrowing the gap towards the $Ω(\logε^{-1})$ lower bound. Our improvement is based on an application of the sunflower lemma previously used by Erdős and Sárközy (1992) in the context of subset sums. Our allocator achieves polylogarithmic overhead only in expectation, and sometimes performs expensive rebuilds. Our second technical result shows that this is necessary: it is impossible to achieve subpolynomial overhead with high probability.

Figures (5)

• Figure 1: An illustration for the proof of \ref{['prop:fewtype']} with two types of object sizes. (i) Initial state at $c=3$, where object $x$ is to be deleted. We find another object $y$ of the same size in level $0$. (ii) Swap $x$ and $y$ before deleting $x$. (iii) Increase the counter to $c=4$, and rebuild levels $2, 1$, and $0$.
• Figure 2: Visualization of two full memory states $\phi,\phi'$ with $M=14$ slots. The unchanged objects in $\phi$ and $\phi'$ (highlighted in dark gray) have sizes $5$ and $3$. The maximal changed intervals between them (highlighted in dashed blue) are $[0,2)$ and $[7,11)$. To transform $\phi$ to $\phi'$, the allocator must incur a total switching cost of at least $\Delta(\phi,\phi')=2+3+1=6$.
• Figure 3: An illustration for the proof of \ref{['prop:warmup']}. Between memory states $\phi_i,\phi_{i+1}$ satisfying the prefix property, every maximal changed interval $[L_k,R_k)$ (marked in dashed blue), except for the rightmost one, permutes the objects inside.
• Figure 4: An illustration for \ref{['lem:finger']} with two memory states $\phi,\phi'$ (corresponding to $S_{3,5}$ and $S_{3,4}$ in \ref{['defn:sizeprof']} respectively). Two maximal changed intervals are marked in dashed blue. The first interval contains objects of sizes $\{a,a,b,c\}$ in both $\phi, \phi'$. The second interval contains the finger objects of both $\phi,\phi'$.
• Figure 5: The rooted tree $\mathcal{T}$ defined in \ref{['defn:sigma']}. The sequence $\sigma_{i,j}$ (shaded in gray) is the tree path from the root node $S_{0,Q}$ to the node $S_{i,j}$. The tree has a spine (shown in blue dashed box) $(S_{0,Q},S_{1,Q},\dots, S_{P,Q})$, and each node on the spine is the root of a branch$(S_{i,Q},S_{i,Q-1},\dots,S_{i,0})$.

Theorems & Definitions (60)

• Theorem 1.1: \ref{['sec:ds']}
• Corollary 1.2
• Theorem 1.3: \ref{['sec:lb']}
• Theorem 1.4: \ref{['sec:lb']}
• Proposition 1.5: based on Farach-ColtonKS24
• proof : Proof of \ref{['prop:fewtype']} (based on Farach-ColtonKS24)
• Proposition 1.6
• proof : Proof of \ref{['prop:warmup']}
• proof
• proof