Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Increasing the Hit Ratio Hurt Cache Throughput? (Long Version)

Ziyue Qiu, Juncheng Yang, Mor Harchol-Balter

TL;DR

It is shown that increasing the hit ratio can actually hurt the throughput (and response time) for many caching algorithms, and the phenomenon of throughput decreasing at higher hit ratios is likely to be more pronounced in future systems, where the trend is towards faster disks and higher numbers of cores per CPU.

Abstract

Software caches are an intrinsic component of almost every computer system. Consequently, caching algorithms, particularly eviction policies, are the topic of many papers. Almost all these prior papers evaluate the caching algorithm based on its hit ratio, namely the fraction of requests that are found in the cache, as opposed to disk. The hit ratio is viewed as a proxy for traditional performance metrics like system throughput or response time. Intuitively it makes sense that higher hit ratio should lead to higher throughput (and lower response time), since more requests are found in the cache (low access time) as opposed to the disk (high access time). This paper challenges this intuition. We show that increasing the hit ratio can actually hurt the throughput (and response time) for many caching algorithms. Our investigation follows a three-pronged approach involving (i) queueing modeling and analysis, (ii) implementation and measurement, and (iii) simulation to validate the accuracy of the queueing model. We also show that the phenomenon of throughput decreasing at higher hit ratios is likely to be more pronounced in future systems, where the trend is towards faster disks and higher numbers of cores per CPU.

Can Increasing the Hit Ratio Hurt Cache Throughput? (Long Version)

TL;DR

It is shown that increasing the hit ratio can actually hurt the throughput (and response time) for many caching algorithms, and the phenomenon of throughput decreasing at higher hit ratios is likely to be more pronounced in future systems, where the trend is towards faster disks and higher numbers of cores per CPU.

Abstract

Software caches are an intrinsic component of almost every computer system. Consequently, caching algorithms, particularly eviction policies, are the topic of many papers. Almost all these prior papers evaluate the caching algorithm based on its hit ratio, namely the fraction of requests that are found in the cache, as opposed to disk. The hit ratio is viewed as a proxy for traditional performance metrics like system throughput or response time. Intuitively it makes sense that higher hit ratio should lead to higher throughput (and lower response time), since more requests are found in the cache (low access time) as opposed to the disk (high access time). This paper challenges this intuition. We show that increasing the hit ratio can actually hurt the throughput (and response time) for many caching algorithms. Our investigation follows a three-pronged approach involving (i) queueing modeling and analysis, (ii) implementation and measurement, and (iii) simulation to validate the accuracy of the queueing model. We also show that the phenomenon of throughput decreasing at higher hit ratios is likely to be more pronounced in future systems, where the trend is towards faster disks and higher numbers of cores per CPU.
Paper Structure (31 sections, 40 equations, 14 figures, 2 tables)

This paper contains 31 sections, 40 equations, 14 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Cache eviction algorithms
  3. The quest for higher hit ratio
  4. A 3-pronged approach to determining if higher hit ratio helps
  5. A. Queueing model for upper bounding throughput
  6. B. Simulation of the queueing model
  7. C. Implementation of the caching system
  8. Why increasing the hit ratio can hurt, in brief
  9. Contributions
  10. Background
  11. A three-pronged approach to LRU
  12. A closed-loop queueing model of LRU
  13. Modeling concurrency in a closed-loop model
  14. Modeling disk access and cache access
  15. Modeling software global list operations
  16. ...and 16 more sections

Figures (14)

  • Figure 1: Throughput under LRU (measured in millions of requests per second) increases as the hit ratio increases initially but then drops when the hit ratio gets high.
  • Figure 2: Queueing model of LRU cache.
  • Figure 3: Results for theory, implementation, and simulation under an LRU cache. The throughput of the LRU cache decreases at higher hit ratios. This trend becomes more pronounced as we move towards lower disk latencies -- from (a) to (b) to (c).
  • Figure 4: Queueing model of FIFO cache.
  • Figure 5: Results for theory, implementation and simulation under a FIFO cache. For all three curves, the throughput of the FIFO cache always increases at higher hit ratios under different disk latencies.
  • ...and 9 more figures