Clock2Q+: A Simple and Efficient Replacement Algorithm for Metadata Cache in VMware vSAN

TL;DR

Metadata caches in storage systems routinely exhibit correlated references that degrade hit ratios when treated as hot blocks. Clock2Q+ fuses Clock2Q and S3-FIFO concepts with a correlation window to filter correlated references, achieving lower miss ratios and scalable performance. It delivers up to 28.5% metadata-miss improvement over S3-FIFO, extends benefits to data traces, and emphasizes production readiness through simple data structures, concurrency safeguards, and online resizing. Implemented in VMware products, Clock2Q+ demonstrates strong practical impact for large-scale storage systems requiring low overhead and easy deployment.

Abstract

Cache replacement algorithms are critical building blocks of storage systems. This paper examines the characteristics of metadata caches and argues that they inherently exhibit correlated references, even when the corresponding data accesses do not contain correlated references. The presence of correlated references reduces the effectiveness of cache replacement algorithms because these references are often mistakenly categorized as hot blocks. Clock2Q+ is specifically designed for metadata caches and has been implemented in vSAN and VDFS, two flagship storage products of VMware by Broadcom. Similar to S3-FIFO, Clock2Q+ uses three queues; however, Clock2Q+ introduces a correlation window in the Small FIFO queue, where blocks in this window do not set the reference bit. This simple enhancement allows Clock2Q+ to outperform state-of-the-art replacement algorithms. Compared to S3-FIFO, the second-best performing algorithm, Clock2Q+ achieves up to a 28.5% lower miss ratio on metadata traces. Clock2Q+ possesses the essential properties required for large-scale storage systems: it has low CPU overhead on cache hits, low memory overhead, scales efficiently to multiple CPUs, and is both easy to tune and implement. Additionally, Clock2Q+ outperforms state-of-the-art cache replacement algorithms on data traces as well.

Figures (14)

• Figure 1: Example B$^+$ tree for mapping LBN to PBN. Lookup LBN $5$ will read metadata block $m1$, $m2$, $m4$ and find LBN $5$ is mapped to PBN $20$.
• Figure 2: Illustration of 2Q cache replacement algorithm.
• Figure 3: Illustration of Clock2Q replacement algorithm.
• Figure 4: Illustration of S3-FIFO replacement algorithm.
• Figure 5: Illustration of Clock2Q+ replacement algorithm.