Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modeling and Optimizing Latency for Delayed Hit Caching with Stochastic Miss Latency

Bowen Jiang, Chaofan Ma

TL;DR

The paper addresses delayed hits under stochastic fetch latencies by deriving exact mean and variance expressions for the aggregate delay when miss times are exponential. It then introduces a variance-aware eviction ranking that blends $E[D_i]$ and $\sigma[D_i]$ to guide cache decisions, using online estimates for residual time and arrival rates. The approach achieves notable latency reductions on synthetic (3\%-30\%) and real-world (1\%-7\%) traces compared with state-of-the-art delayed-hit methods, demonstrating the practical impact of accounting for miss-latency randomness. This work advances caching theory and practice by integrating stochastic latency into the analysis and eviction policy design, improving user-perceived performance in high-throughput systems.

Abstract

Caching is crucial for system performance, but the delayed hit phenomenon, where requests queue during lengthy fetches after a cache miss, significantly degrades user-perceived latency in modern high-throughput systems. While prior works address delayed hits by estimating aggregate delay, they universally assume deterministic fetch latencies. This paper tackles the more realistic, yet unexplored, scenario where fetch latencies are stochastic. We present, to our knowledge, the first theoretical analysis of delayed hits under this condition, deriving analytical expressions for both the mean and variance of the aggregate delay assuming exponentially distributed fetch latency. Leveraging these insights, we develop a novel variance-aware ranking function tailored for this stochastic setting to guide cache eviction decisions more effectively. The simulations on synthetic and real-world datasets demonstrate that our proposed algorithm significantly reduces overall latency compared to state-of-the-art delayed-hit strategies, achieving a $3\%-30\%$ reduction on synthetic datasets and approximately $1\%-7\%$ reduction on real-world traces.

Modeling and Optimizing Latency for Delayed Hit Caching with Stochastic Miss Latency

TL;DR

The paper addresses delayed hits under stochastic fetch latencies by deriving exact mean and variance expressions for the aggregate delay when miss times are exponential. It then introduces a variance-aware eviction ranking that blends and to guide cache decisions, using online estimates for residual time and arrival rates. The approach achieves notable latency reductions on synthetic (3\%-30\%) and real-world (1\%-7\%) traces compared with state-of-the-art delayed-hit methods, demonstrating the practical impact of accounting for miss-latency randomness. This work advances caching theory and practice by integrating stochastic latency into the analysis and eviction policy design, improving user-perceived performance in high-throughput systems.

Abstract

Caching is crucial for system performance, but the delayed hit phenomenon, where requests queue during lengthy fetches after a cache miss, significantly degrades user-perceived latency in modern high-throughput systems. While prior works address delayed hits by estimating aggregate delay, they universally assume deterministic fetch latencies. This paper tackles the more realistic, yet unexplored, scenario where fetch latencies are stochastic. We present, to our knowledge, the first theoretical analysis of delayed hits under this condition, deriving analytical expressions for both the mean and variance of the aggregate delay assuming exponentially distributed fetch latency. Leveraging these insights, we develop a novel variance-aware ranking function tailored for this stochastic setting to guide cache eviction decisions more effectively. The simulations on synthetic and real-world datasets demonstrate that our proposed algorithm significantly reduces overall latency compared to state-of-the-art delayed-hit strategies, achieving a reduction on synthetic datasets and approximately reduction on real-world traces.

Paper Structure

This paper contains 14 sections, 2 theorems, 20 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Problem Definition and Motivation
  3. Problem Formulation
  4. The Aggregate Delay and Motivation
  5. Characteristics of Aggregate Delay
  6. The distribution of aggregate delay
  7. The mean and variance of aggregate delay
  8. Our algorithm
  9. Simulation Results
  10. Methodology
  11. Synthetic dataset
  12. Real world traces
  13. Sensitivity Analysis
  14. Conclusion

Key Result

Theorem 1

Assume object $i$ arrives according to a Poisson process with rate $\lambda_i$ and the miss latency is $z_i$. The mean and variance of $D_i$ are:

Figures (5)

  • Figure 1: A toy example to show the importance of variance in cache decision.
  • Figure 2: Comparison of latency improvement between our algorithm and SOTAs under the synthetic dataset($C=500\text{MB}$).
  • Figure 3: The content popularity and average interval-time distributions of the four real-world traces.
  • Figure 4: The impact of hyperparameters on latency improvement.
  • Figure 5: Comparison of latency improvement between our algorithm and SOTAs using a 256GB cache.

Theorems & Definitions (5)

  • Remark 1
  • Remark 2
  • Theorem 1: in VA
  • Theorem 2
  • Remark 3