DumpKV: Learning based lifetime aware garbage collection for key value separation in LSM-tree

TL;DR

The paper tackles the persistent challenge of high write amplification in LSM-tree KV stores with KV separation by introducing DumpKV, a learning-based lifetime-aware garbage collection framework. DumpKV predicts per-key lifetimes using features derived from past writes (Delta intervals and EDWCs) and dynamically adjusts lifetime thresholds for value files, performing feature collection during L0-L1 compaction to minimize write disruption. A lightweight gradient boosting model is trained online to classify keys by remaining lifetime, guiding where to relocate values and how to schedule GC, with crash-consistent design for persistence of essential state. Empirical results on a RocksDB-based prototype show DumpKV reduces total write size by 38%–73% compared to existing KV separation implementations, with modest feature storage overhead and competitive total size and throughput across varying workloads and value sizes.

Abstract

Key\-value separation is used in LSM\-tree to stored large value in separate log files to reduce write amplification, but requires garbage collection to garbage collect invalid values. Existing garbage collection techniques in LSM\-tree typically adopt static parameter based garbage collection to garbage collect obsolete values which struggles to achieve low write amplification and it's challenging to find proper parameter for garbage collection triggering. In this work we introduce DumpKV, which introduces learning based lifetime aware garbage collection with dynamic lifetime adjustment to do efficient garbage collection to achieve lower write amplification. DumpKV manages large values using trained lightweight model with features suitable for various application based on past write access information of keys to give lifetime prediction for each individual key to enable efficient garbage collection. To reduce interference to write throughput DumpKV conducts feature collection during L0\-L1 compaction leveraging the fact that LSM\-tree is small under KV separation. Experimental results show that DumpKV achieves lower write amplification by 38\%\-73\% compared to existing key\-value separation garbage collection LSM\-tree stores with small feature storage overhead.

Figures (11)

• Figure 1: LSM-tree and KV separation
• Figure 2: LSM-tree and KV separation
• Figure 3: DumpKV architecture
• Figure 4: Feature storage format in LSM-tree
• Figure 5: DumpKV garbage collection process