Dynamic Content Caching with Waiting Costs via Restless Multi-Armed Bandits

TL;DR

This work addresses dynamic content caching with AoV-based ageing and waiting costs by modeling each content as an arm in a Restless Multi-Armed Bandit (RMAB). It develops a Whittle-index based policy with explicit index expressions, and proves asymptotic optimality for large content sets with capacity scaling. A complete solution is provided for the infinite-cache case, while a finite-cache policy leverages indexability and per-content indices to govern eviction and fetching. Numerical results show the proposed policy closely matches the relaxed RMAB lower bound and significantly outperforms greedy and static push-pull baselines, with the waiting option reducing costs in practice.

Abstract

We consider a system with a local cache connected to a backend server and an end user population. A set of contents are stored at the the server where they continuously get updated. The local cache keeps copies, potentially stale, of a subset of the contents. The users make content requests to the local cache which either can serve the local version if available or can fetch a fresh version or can wait for additional requests before fetching and serving a fresh version. Serving a stale version of a content incurs an age-of-version(AoV) dependent ageing cost, fetching it from the server incurs a fetching cost, and making a request wait incurs a per unit time waiting cost. We focus on the optimal actions subject to the cache capacity constraint at each decision epoch, aiming at minimizing the long term average cost. We pose the problem as a Restless Multi-armed Bandit(RMAB) Problem and propose a Whittle index based policy which is known to be asymptotically optimal. We explicitly characterize the Whittle indices. We numerically evaluate the proposed policy and also compare it to a greedy policy. We show that it is close to the optimal policy and substantially outperforms the exising policies.

Figures (5)

• Figure 1: Possible actions for the contents $\{r\}\cup\{\mathcal{E}_k\}$. For instance, Action $2$ is applicable for all the contents in $\{r\}\cup\{\mathcal{E}_k\}$, however, it means wait for content $r$ and evict for the contents in $\mathcal{E}_k\setminus \{r\}$.
• Figure 2: Possible actions for the tagged content with holding cost. Here we have additional actions, Action $3$ denoting fetch, serve and evict and Action $4$ denoting serve the cached version and evict, compared to Figure \ref{['fig:finite_capacity_with_queue']}.
• Figure 3: Optimal policy structure with respect to $C_h$ for states $(0,\tau,1,0)$ and $(Q,0,1)$; $\tau^{\ast}, Q^{\ast}, \bar{\tau}, \bar{Q}, \hat{Q}$, and $I$ are as in Theorem \ref{['thm:fin_cap_wait']}.
• Figure 4: Performance of the Whittle index based policy. For each cache size, its cost is close to the optimal value of the relaxed RMAB problem and is much smaller than the cost of a Myopic and push-pull based caching policy 10452408.
• Figure 5: Impact of $c_w$ on average cost and waiting time. $(a)$ The average cost increases as $c_w$ increases but it becomes oblivious to $c_w$ beyond $c_w = 0.1$. $(b)$ The average wait time reduces as $c_w$ increases is very close to zero for $c_w \geq 0.1$.

Theorems & Definitions (16)

• Theorem 1
• Proposition 1
• Theorem 2
• Lemma 3
• Lemma 4
• Theorem 5
• Remark 1: Asymptotic optimality
• Remark 2: Impatient requests
• Lemma 6
• Lemma 7