Tidehunter: Large-Value Storage With Minimal Data Relocation

TL;DR

Tidehunter tackles the dominant cost of LSM-tree based stores—value compaction—by making the Write-Ahead Log the permanent storage for values and using small, lazily flushed indices. It introduces an optimistic, on-disk index that exploits uniform key distributions for fast single-roundtrip lookups, and a relocation mechanism to reclaim space without blocking writes. The system achieves up to $8.4\times$ higher write throughput than RocksDB on large values, and delivers strong improvements in existence checks and read latency under skewed workloads, with production validation in the Sui blockchain showing stable performance under heavy load. By integrating append-only WAL with sharded indices and asynchronous durability, Tidehunter demonstrates a practical path to scalable, high-throughput storage for content-addressable data, deduplication, and blockchain validation workloads where values dominate storage movement.

Abstract

Log-Structured Merge-Trees (LSM-trees) dominate persistent key-value storage but suffer from high write amplification from 10x to 30x under random workloads due to repeated compaction. This overhead becomes prohibitive for large values with uniformly distributed keys, a workload common in content-addressable storage, deduplication systems, and blockchain validators. We present Tidehunter, a storage engine that eliminates value compaction by treating the Write-Ahead Log (WAL) as permanent storage rather than a temporary recovery buffer. Values are never overwritten; and small, lazily-flushed index tables map keys to WAL positions. Tidehunter introduces (a) lock-free writes that saturate NVMe drives through atomic allocation and parallel copying, (b) an optimistic index structure that exploits uniform key distributions for single-roundtrip lookups, and (c) epoch-based pruning that reclaims space without blocking writes. On a 1 TB dataset with 1 KB values, Tidehunter achieves 830K writes per second, that is 8.4x higher than RocksDB and 2.9x higher than BlobDB, while improving point queries by 1.7x and existence checks by 15.6x. We validate real-world impact by integrating Tidehunter into Sui, a high-throughput blockchain, where it maintains stable throughput and latency under loads that cause RocksDB-backed validators to collapse. Tidehunter is production-ready and is being deployed in production within Sui.

Figures (13)

• Figure 1: Mixed workload throughput (50% reads/writes) vs. value size, with skewed (Zipf $\theta=2$) and homogeneous (Zipf $\theta=0$) access patterns.
• Figure 2: Design evolution of Tidehunter. Each step addresses a limitation of the previous design: (1) append-only log provides fast writes but slow reads; (2) in-memory index enables fast lookups; (3) persisted index supports datasets larger than memory; (4) relocation reclaims space without blocking writes; (5) snapshots enable fast recovery; (6) sharding enables concurrent operations.
• Figure 3: High-level architecture of Tidehunter.
• Figure 4: Write path separating fast synchronous operations (solid) from background durability (dotted).
• Figure 5: Read path showing the tiered lookup strategy. The snapshot component is omitted for clarity.