Stone Soup Multi-Target Tracking Feature Extraction For Autonomous Search And Track In Deep Reinforcement Learning Environment

TL;DR

The paper tackles autonomous sensor management for heterogeneous sensing in future aerial systems by leveraging Stone Soup as a multi-target tracking feature extractor within a Gymnasium-enabled RL loop. It introduces a framework to embed Stone Soup components into Stable Baselines3 workflows, enabling MT trackers and track lists to form informative observations for RL agents. Through a demonstrative AESA radar search-and-track problem, the study shows that RL agents using Stone Soup-based features can outperform basic sensor policies, with BiGRU+MHSA architectures delivering the strongest performance and GOSPA reduction. The work demonstrates the practicality of integrating track-based feature extraction into RL training and identifies avenues for future enhancements, such as multi-agent setups and trajectory-based tracking methods. Overall, this approach offers a scalable path to improving autonomy and efficiency in complex sensing environments.

Abstract

Management of sensing resources is a non-trivial problem for future military air assets with future systems deploying heterogeneous sensors to generate information of the battlespace. Machine learning techniques including deep reinforcement learning (DRL) have been identified as promising approaches, but require high-fidelity training environments and feature extractors to generate information for the agent. This paper presents a deep reinforcement learning training approach, utilising the Stone Soup tracking framework as a feature extractor to train an agent for a sensor management task. A general framework for embedding Stone Soup tracker components within a Gymnasium environment is presented, enabling fast and configurable tracker deployments for RL training using Stable Baselines3. The approach is demonstrated in a sensor management task where an agent is trained to search and track a region of airspace utilising track lists generated from Stone Soup trackers. A sample implementation using three neural network architectures in a search-and-track scenario demonstrates the approach and shows that RL agents can outperform simple sensor search and track policies when trained within the Gymnasium and Stone Soup environment.

Figures (8)

• Figure 1: Reinforcement learning training process adapted from sutton1998reinforcement. The agent interacts with the environment through a defined action set $\boldsymbol{a}_{t}$ based on the observation $\boldsymbol{s}_{t}$ and the reward $r_t$.
• Figure 2: RL problem structure utilising Gymnasium environment containing Stone Soup components with suitable wrapper functions to integrate agent actions $\boldsymbol{a}_{k}$ with the sensor.
• Figure 3: Stone Soup MTT component framework as provided in barr2022stone.
• Figure 4: Gymnasium towers2024gymnasium environment construction utilising Stone Soup components for ground truth generation, multi-target tracker implementation and metrics generation.
• Figure 5: Search region with $N_{a}\times{}N_{a}$ overlapping cells to ensure 100% coverage of the space. Agent action $\boldsymbol{a}_{k}$ is mapped to a cell with defined azimuth $\psi$ and elevation $\theta$ angles.