An Index Policy Based on Sarsa and Q-learning for Heterogeneous Smart Target Tracking

TL;DR

This paper tackles non-myopic radar scheduling for multiple smart targets in an active-passive radar network by formulating the problem as a restless multi-armed bandit with unknown, heterogeneous target dynamics. It introduces the ISQ policy, a unified online index-policy that combines forward Sarsa and backward Q-learning to learn per-target Q-values and state-dependent indices $\\lambda^{n*}(X^n_t) = Q^{n*}(X^n_t,1) - Q^{n*}(X^n_t,0)$, and selects the top-$K$ targets to maximize discounted long-term rewards under the constraint $\sum a^n_t = K$. Through simulations on circulant dynamics, homogeneous, and heterogeneous target settings, ISQ demonstrates faster convergence and superior performance relative to WIQL, AB, and Greedy, while approaching the Whittle index policy with full knowledge as a near-optimal benchmark. The work provides a scalable, online framework for robust radar beam scheduling in APRN where transition models are unknown, enabling improved long-term tracking in realistic, dynamic environments. The combination of Sarsa forward learning and Q-learning backward updates yields improved convergence and adaptability for per-target learning, making ISQ practical for real-time radar resource allocation. All mathematical notation is presented with explicit $...$ delimiters to ensure precise, machine-readable representations of the underlying algorithms.

Abstract

In solving the non-myopic radar scheduling for multiple smart target tracking within an active and passive radar network, we need to consider both short-term enhanced tracking performance and a higher probability of target maneuvering in the future with active tracking. Acquiring the long-term tracking performance while scheduling the beam resources of active and passive radars poses a challenge. To address this challenge, we model this problem as a Markov decision process consisting of parallel restless bandit processes. Each bandit process is associated with a smart target, of which the estimation state evolves according to different discrete dynamic models for different actions - whether or not the target is being tracked. The discrete state is defined by the dynamic mode. The problem exhibits the curse of dimensionality, where optimal solutions are in general intractable. We resort to heuristics through the famous restless multi-armed bandit techniques. It follows with efficient scheduling policies based on the indices that are real numbers representing the marginal rewards of taking different actions. For the inevitable practical case with unknown transition matrices, we propose a new method that utilizes the forward Sarsa and backward Q-learning to approximate the indices through adapting the state-action value functions, or equivalently the Q-functions, and propose a new policy, namely ISQ, aiming to maximize the long-term tracking rewards. Numerical results demonstrate that the proposed ISQ policy outperforms conventional Q-learning-based methods and rapidly converges to the well-known Whittle index policy with revealed state transition models, which is considered the benchmark.

Figures (8)

• Figure 1: MDPs of smart targets under actions. (a) The tendency for maneuvering under action $a=1$. (b) The tendency for non-maneuvering under action $a=0$.
• Figure 2: The framework of Reinforcement Learning.
• Figure 3: The framework of the index policy based on reinforcement learning.
• Figure 4: The proposed framework of radars scheduling in the APRN.
• Figure 5: Average cumulative reward for the circulant dynamics problem. (a) $N=5$, $K=1$. (b) $N=100$, $K=20$.