WritePolicyBench: Benchmarking Memory Write Policies under Byte Budgets

TL;DR

WritePolicyBench tackles memory-constrained streaming with external memory by defining a byte-budgeted write policy problem under distribution drift. It provides a deterministic benchmark with drift-regime task generators, an explicit WRITE/EXPIRE/MERGE/SKIP action interface, a byte-accurate cost model, and standardized metrics that jointly evaluate task success and budget efficiency through final retained sets $W$ relative to drift labels $R$ and utilities $u_t$, including an oracle-based regret $\max(0, U^\star(B) - U)$. The study presents unprivileged and privileged evaluation tracks with several baselines, revealing insights such as drift coverage saturating with budget, potential precision degradation at high budgets, and nuanced effects of MERGE-based compression. Overall, the framework offers a reproducible, fine-grained substrate for comparing write policies under tight memory budgets and non-stationary data, guiding memory-management design in external-memory systems and drift-prone applications; it also highlights limitations and avenues for learned policies and more realistic drift scenarios. $W$, $R$, $u_t$, $B$, and $U^ abla(B)$ are central quantities, enabling precise, knapsack-like budgeting and objective-driven evaluation under drift.

Abstract

We introduce WritePolicyBench, a benchmark for evaluating memory write policies: decision rules that choose what to store, merge, and evict under a strict byte budget while processing a stream with document/API drift. The benchmark provides (i) task generators with controlled non-stationarity, (ii) an explicit action interface for external memory, (iii) a byte-accurate cost model, and (iv) standardized metrics that measure both task success and budget efficiency.

Figures (1)

• Figure 1: Budget curves for representative regimes (privileged track). Shaded regions indicate $\pm 1$ standard error over 10 episodes.