Adaptive Scheduling: A Reinforcement Learning Whittle Index Approach for Wireless Sensor Networks

TL;DR

This work tackles goal-oriented sensor scheduling under limited communication by cast­ing the problem as a Restless Multi-Armed Bandit (RMAB) and learning Whittle indices online. It introduces WIQL-UCB, a hyperparameter-free, per-arm Q-learning approach that uses Upper Confidence Bound exploration to estimate Whittle indices and select the best $M$ arms at each step, achieving near-optimal performance with strong scalability. The method leverages edge mining to transform raw sensor data into informative, edge-processed states, enabling effective AoII-based scheduling without prior model knowledge. Experimental results across circulant dynamics, processing updates, and real-world environmental monitoring demonstrate superior memory efficiency, fast per-decision runtimes, and robust performance against both Whittle-based and non-Whittle baselines, highlighting practical impact for constrained wireless sensor networks.

Abstract

We propose a reinforcement learning based scheduling framework for Restless Multi-Armed Bandit (RMAB) problems, centred on a Whittle Index Q-Learning policy with Upper Confidence Bound (UCB) exploration, referred to as WIQL-UCB. Unlike existing approaches that rely on fixed or adaptive epsilon-greedy strategies and require careful hyperparameter tuning, the proposed method removes problem-specific tuning and is therefore more generalisable across diverse RMAB settings. We evaluate WIQL-UCB on standard RMAB benchmarks and on a practical sensor scheduling application based on the Age of Incorrect Information (AoII), using an edge-based state estimation scheme that requires no prior knowledge of system dynamics. Experimental results show that WIQL-UCB achieves near-optimal performance while significantly improving computational and memory efficiency. For a representative problem size of N = 15 and M = 3, the proposed method requires only around 600 bytes of memory, compared with several kilobytes for tabular Q-learning and hundreds of kilobytes to megabytes for deep reinforcement learning baselines. In addition, WIQL-UCB achieves sub-millisecond per-decision runtimes and is several times faster than deep reinforcement learning approaches, while maintaining competitive performance. Overall, these results demonstrate that WIQL-UCB consistently outperforms both non-Whittle-based and Whittle-index learning baselines across a wide range of RMAB settings.

Figures (13)

• Figure 1: Centralised sensor scheduling framework considered in this work. Each sensor continuously monitors the underlying physical process and performs local sensing. Raw measurements are locally transformed into state estimates, but are not transmitted unless the sensor is scheduled. At each time step, a central scheduler selects $M$ out of $N$ sensors subject to channel constraints. The scheduler runs the - algorithm, updating its scheduling policy based on the received state updates to optimise the chosen performance objective.
• Figure 2: Comparison of average rewards for different scheduling techniques across varying values of $M$. Each method is evaluated against the oracle optimal policy, which assumes full knowledge of the node-side transition dynamics. In settings with fewer arms, the performance of all policies is relatively similar. However, as the $N/M$ ratio increases indicating a lower activation budget relative to the number of arms the performance gaps widen. In particular, - demonstrates superior scalability, achieving performance closest to the oracle benchmark as resource constraints become more pronounced. Experimental results are averaged over 10 simulation runs
• Figure 3: Comparison of average rewards for different scheduling techniques in static (top row) and dynamic (bottom row) environments, across varying values of $M$ and $N$. - consistently outperforms the other strategies, particularly as system constraints increase. In contrast, -Fu and -AB struggle to adapt in dynamic environments due to their reliance on outdated transition probability estimates and slower adaptation to changing system dynamics.
• Figure 4: Comparison of rewards for various scheduling techniques under different values of $M$. - consistently achieves the highest reward across all settings. The performance gap is most pronounced when the ratio $N/M$ is high, and gradually decreases as the number of activations $M$ increases.
• Figure 5: Node pulling comparison across categories. - activates nodes in Category C most frequently, aligning activation priority with variation level. In contrast, and allocate pulls uniformly across categories, ignoring differences in variation.

Theorems & Definitions (3)

• Definition 1: Indexability
• Theorem 1
• Theorem 2