Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VA-CDH: A Variance-Aware Method to Optimize Latency for Caching with Delayed Hits

Bowen Jiang, Chaofan Ma, Duo Wang

TL;DR

The paper tackles latency in caching with delayed hits by revealing that the variance of aggregate delay, not just its mean, critically affects performance. It derives the Poisson-based distribution of the aggregate delay $D_i$ and proposes VA-CDH, a variance-aware eviction policy that combines the mean $E[D_i]$ and standard deviation $σ[D_i]$ in a mean-plus-variance ranking. VA-CDH uses a sliding-window online estimator for per-object arrival rates and residual times to compute eviction priorities and adapt to non-stationary workloads. Empirical results on synthetic and real traces show consistent latency reductions of about $1\%-6\%$ over state-of-the-art delayed-hit algorithms, demonstrating practical benefits for CDN and MEC environments.

Abstract

Caches are fundamental to latency-sensitive systems like Content Delivery Networks (CDNs) and Mobile Edge Computing (MEC). However, the delayed hit phenomenon where multiple requests for an object occur during its fetch from the remote server after a miss significantly inflates user-perceived latency. While recent algorithms acknowledge delayed hits by estimating the resulting aggregate delay, they predominantly focus on its mean value. We identify and demonstrate that such approaches are insufficient, as the real aggregate delay frequently exhibits substantial variance in the true production system, leading to suboptimal latency performance when ignored. Thus, we propose VA-CDH, a variance-aware method to optimize latency for caching with delayed hits. It employs a novel ranking function that explicitly incorporates both the empirically estimated mean and standard deviation of aggregate delay, allowing caching decisions to account for its variation. We derive the analytical distribution of aggregate delay under Poisson arrivals as a theoretical contribution, offering more statistical insight beyond the mean value. Through the simulations conducted on synthetic and real-world datasets, we show that VA-CDH reduces the total latency by 1%-6% approximately compared to state-of-the-art algorithms.

VA-CDH: A Variance-Aware Method to Optimize Latency for Caching with Delayed Hits

TL;DR

The paper tackles latency in caching with delayed hits by revealing that the variance of aggregate delay, not just its mean, critically affects performance. It derives the Poisson-based distribution of the aggregate delay and proposes VA-CDH, a variance-aware eviction policy that combines the mean and standard deviation in a mean-plus-variance ranking. VA-CDH uses a sliding-window online estimator for per-object arrival rates and residual times to compute eviction priorities and adapt to non-stationary workloads. Empirical results on synthetic and real traces show consistent latency reductions of about over state-of-the-art delayed-hit algorithms, demonstrating practical benefits for CDN and MEC environments.

Abstract

Caches are fundamental to latency-sensitive systems like Content Delivery Networks (CDNs) and Mobile Edge Computing (MEC). However, the delayed hit phenomenon where multiple requests for an object occur during its fetch from the remote server after a miss significantly inflates user-perceived latency. While recent algorithms acknowledge delayed hits by estimating the resulting aggregate delay, they predominantly focus on its mean value. We identify and demonstrate that such approaches are insufficient, as the real aggregate delay frequently exhibits substantial variance in the true production system, leading to suboptimal latency performance when ignored. Thus, we propose VA-CDH, a variance-aware method to optimize latency for caching with delayed hits. It employs a novel ranking function that explicitly incorporates both the empirically estimated mean and standard deviation of aggregate delay, allowing caching decisions to account for its variation. We derive the analytical distribution of aggregate delay under Poisson arrivals as a theoretical contribution, offering more statistical insight beyond the mean value. Through the simulations conducted on synthetic and real-world datasets, we show that VA-CDH reduces the total latency by 1%-6% approximately compared to state-of-the-art algorithms.
Paper Structure (15 sections, 3 theorems, 11 equations, 4 figures)

This paper contains 15 sections, 3 theorems, 11 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Problem Definition and Motivation
  3. Setting
  4. The Aggregate Delay and Motivation
  5. Characteristics of Aggregate Delay
  6. The distribution of aggregate delay
  7. The mean and variance of aggregate dealy
  8. Variance-Aware Methond
  9. Variance-Aware Rank Function
  10. VA-CDH Algorithm
  11. Simulation Results
  12. Synthetic dataset
  13. Real world traces
  14. Sensitivity Analysis
  15. Conclusion

Key Result

Theorem 1

Assume object $i$ arrives according to a Poisson process with rate $\lambda_i$, and the fetch time upon a miss is a constant $z_i$. The Probability Density Function (PDF) of the aggregate delay $D_i$ for object $i$ is a mixture distribution given by: where $\delta(\cdot)$ is the Dirac delta function, $N_i^z$ is the random variable for the number of arrivals during the fetch period of length $z_i$

Figures (4)

  • Figure 1: The estimated aggregate delay vs. real aggregate delay calculated by equation \ref{['Aggdelay']}.
  • Figure 2: Comparison of latency improvement of VA-CDH and SOTAs under the synthetic dataset ($C=500\text{MB}$).
  • Figure 3: Comparison of latency improvement of VA-CDH and SOTAs using a 256GB cache.
  • Figure 4: The impact of hyperparameters on latency improvement. (a) fixed $S=10K$, (b) fixed $\omega=1$.

Theorems & Definitions (5)

  • Theorem 1
  • Lemma 1
  • Theorem 2
  • Remark 1
  • Remark 2