Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Whittle's index-based age-of-information minimization in multi-energy harvesting source networks

Akanksha Jaiswal, Arpan Chattopadhyay

TL;DR

This work tackles AoI minimization in a network of multiple energy-harvesting sources sharing a single probing opportunity. By formulating a CMDP and applying Lagrangian relaxation, the authors decouple the problem into per-source subproblems and derive a Whittle-index and threshold-based policy (WITS3) in which the source with the highest index is probed and sampled only when the channel quality exceeds a state-dependent threshold. The paper proves indexability and structures the sampling policy via a threshold on channel quality, and introduces Q-WITS3 to learn Whittle indices and policies when channel and EH statistics are unknown, using two-timescale asynchronous Q-learning. Numerical results show WITS3 outperforms two greedy baselines, and Q-WITS3 asymptotically matches WITS3, illustrating practical efficacy in both known and unknown environments. The approach offers a scalable, near-optimal solution for AoI minimization in EH networks with probing capability and has potential extensions to more complex topologies and dynamic channel models.

Abstract

We consider the problem of source sampling and transmission scheduling for age-of-information minimization in a system consisting of multiple energy harvesting (EH) sources and a sink node. At each time, one of the sources is selected by the scheduler and the quality of its channel to the sink is measured. This probed channel quality is then used to decide whether a source will sample an observation and transmit the packet to the sink in that time slot. We formulate this problem as a constrained Markov decision process (CMDP) assuming i.i.d. energy arrival and channel fading processes, and relax it using a Lagrange multiplier. We apply a near optimal Whittle's index policy to decide the node to be probed. Next, for the probed node, we derive an optimal threshold policy, which recommends source sampling and observation transmission from the probed source only when the measured channel quality is above a threshold. Our proposed policy is called Whittle's index and threshold based source scheduling and sampling (WITS3) policy. However, in order to calculate Whittle's indices, one must be aware of the underlying processes' transition matrices, which are occasionally concealed from the scheduler. Therefore, we further propose a variant Q-WITS3 of WITS3 based on Q-learning assisted by two timescale asynchronous stochastic approximation, which seeks to learn Whittle's indices and optimal policies for the case with unknown channel states and EH characteristics. Numerical results demonstrate the efficacy of our algorithms over two baseline policies.

Whittle's index-based age-of-information minimization in multi-energy harvesting source networks

TL;DR

This work tackles AoI minimization in a network of multiple energy-harvesting sources sharing a single probing opportunity. By formulating a CMDP and applying Lagrangian relaxation, the authors decouple the problem into per-source subproblems and derive a Whittle-index and threshold-based policy (WITS3) in which the source with the highest index is probed and sampled only when the channel quality exceeds a state-dependent threshold. The paper proves indexability and structures the sampling policy via a threshold on channel quality, and introduces Q-WITS3 to learn Whittle indices and policies when channel and EH statistics are unknown, using two-timescale asynchronous Q-learning. Numerical results show WITS3 outperforms two greedy baselines, and Q-WITS3 asymptotically matches WITS3, illustrating practical efficacy in both known and unknown environments. The approach offers a scalable, near-optimal solution for AoI minimization in EH networks with probing capability and has potential extensions to more complex topologies and dynamic channel models.

Abstract

We consider the problem of source sampling and transmission scheduling for age-of-information minimization in a system consisting of multiple energy harvesting (EH) sources and a sink node. At each time, one of the sources is selected by the scheduler and the quality of its channel to the sink is measured. This probed channel quality is then used to decide whether a source will sample an observation and transmit the packet to the sink in that time slot. We formulate this problem as a constrained Markov decision process (CMDP) assuming i.i.d. energy arrival and channel fading processes, and relax it using a Lagrange multiplier. We apply a near optimal Whittle's index policy to decide the node to be probed. Next, for the probed node, we derive an optimal threshold policy, which recommends source sampling and observation transmission from the probed source only when the measured channel quality is above a threshold. Our proposed policy is called Whittle's index and threshold based source scheduling and sampling (WITS3) policy. However, in order to calculate Whittle's indices, one must be aware of the underlying processes' transition matrices, which are occasionally concealed from the scheduler. Therefore, we further propose a variant Q-WITS3 of WITS3 based on Q-learning assisted by two timescale asynchronous stochastic approximation, which seeks to learn Whittle's indices and optimal policies for the case with unknown channel states and EH characteristics. Numerical results demonstrate the efficacy of our algorithms over two baseline policies.
Paper Structure (14 sections, 2 theorems, 24 equations, 5 figures, 1 algorithm)

This paper contains 14 sections, 2 theorems, 24 equations, 5 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. System model
  3. Problem formulation and proposed policy
  4. MDP Formulation
  5. Constrained MDP, Lagrange Multiplier and Whittle Index
  6. Bellman Equation
  7. Policy structure
  8. Q-Learning for an unknown environment
  9. Optimality equation for decoupled subproblems expressed as Q-function
  10. Q-Learning algorithm for decoupled subproblems
  11. Simulation results
  12. Multiple sources with known system parameters
  13. Q-WITS3 for unknown system parameters
  14. Conclusion

Key Result

Lemma 1

For each source $i$, $J^{*}_{i}(E_{i}, K_{i})$ is increasing in $K_{i}$ and $W^{*}_{i}(E_{i},K_{i},C(i, \cdot)$ is decreasing in $p(C(i,\cdot))$.

Figures (5)

  • Figure 1: Remote sensing system with multiple EH sources.
  • Figure 2: Multiple sources ($N=3$): (a) Variation of $\mathcal{WI}_i$ with $K_i$ for a fixed $E_i$ and (b) Variation of $\mathcal{WI}_i$ with $E_i$ for a fixed $K_i$ where $i \in \{1,2,3\}$.
  • Figure 3: Variation of $p_{th}(E_i,K_i)$ with $E_i$: (a) $\hat{\mu}= 2$, (b) $\hat{\mu}= 4$.
  • Figure 4: Comparison among WITS3, GMA-R and GME-R.
  • Figure 5: Time performance comparison among proposed Q-WITS3 algorithm, the optimal WITS3 policy, and random policy.

Theorems & Definitions (7)

  • Definition 1
  • Conjecture 1
  • Remark 1
  • Conjecture 2
  • Lemma 1
  • Theorem 1
  • proof