A Reinforcement Learning Approach to Sensing Design in Resource-Constrained Wireless Networked Control Systems

TL;DR

The paper addresses sensing design in resource-constrained wireless networked control systems where sensors must choose between raw versus processed transmissions for a dynamical state $x_k$ with estimator covariance $P_k$. It introduces a processing-network model and casts the objective as minimizing the time-averaged $\mathrm{trace}(P_k)$ using data $\mathcal{Y}_k^{\pi}$ under sensing policies $\pi_i$, implemented via a Kalman predictor. To manage computational complexity, the authors adopt a model-free reinforcement learning approach—specifically Q-learning—in a homogeneous-sensors setting, discretizing state and action spaces. Numerical experiments with four drones motivated by Internet of Drones show that the learned policy outperforms static all-raw or all-processed strategies, revealing nontrivial, time-varying sensing schedules. The work highlights a scalable path to co-design sensing, computation, and communication in edge/fog-enabled networked control systems, with potential extensions to heterogeneous networks.

Abstract

In this paper, we consider a wireless network of smart sensors (agents) that monitor a dynamical process and send measurements to a base station that performs global monitoring and decision-making. Smart sensors are equipped with both sensing and computation, and can either send raw measurements or process them prior to transmission. Constrained agent resources raise a fundamental latency-accuracy trade-off. On the one hand, raw measurements are inaccurate but fast to produce. On the other hand, data processing on resource-constrained platforms generates accurate measurements at the cost of non-negligible computation latency. Further, if processed data are also compressed, latency caused by wireless communication might be higher for raw measurements. Hence, it is challenging to decide when and where sensors in the network should transmit raw measurements or leverage time-consuming local processing. To tackle this design problem, we propose a Reinforcement Learning approach to learn an efficient policy that dynamically decides when measurements are to be processed at each sensor. Effectiveness of our proposed approach is validated through a numerical simulation with case study on smart sensing motivated by the Internet of Drones.

Figures (4)

• Figure 1: Data are received at the base station after delay induced by computation (rectangular blocks) and communication (dashed arrows). Local computation at the $i$th sensor is ruled by sensing policy $\pi_{i}$. Given sensing decision $\pi_{i,k_1} = 1$, the sample collected at time $k_1$ is processed (processing delay $\tau_ {{i,\text{proc}}}$), transmitted at time $k_2 = k_1 + \tau_ {{i,\text{proc}}}$ (communication delay $\delta_ {{i,\text{proc}}}$), and received at the base station at time $k_2 + \delta_ {{i,\text{proc}}} = k_1 + \Delta_ {{i,\text{proc}}}$.
• Figure 2: Homogeneous sensors ruled by policy $\pi_\text{hom}$. As decision $\pi_{\text{hom},\ell}$ is communicated, the $i$th sensor disregards current measurement (red cross) and immediately switches to raw mode, acquiring a new sample.
• Figure 3: Time evolution of $\mathrm{trace}\left(P_k\right)$: Q-learning vs. common (static) policies. Windows are marked by vertical lines at times $k = 0, 50, 100,\dots$
• Figure 4: MA of $\mathrm{trace}\left(P_k\right)$: Q-learning vs. common static policies.

Theorems & Definitions (6)

• Remark 1: Multiple processing modes
• Definition 1: Available sensory data
• Remark 2: Complexity with multiple processing modes
• Definition 2: Homogeneous sensing policy
• Remark 3: Heterogeneous sensors
• Remark 4: Nontrivial optimal design