Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recurrent Auto-Encoders for Enhanced Deep Reinforcement Learning in Wilderness Search and Rescue Planning

Jan-Hendrik Ewers, David Anderson, Douglas Thomson

TL;DR

The paper tackles WiSAR planning over vast terrains where full coverage is impractical by introducing a Recurrent Autoencoder (LSTMAE) that preprocesses temporal observations into a compact latent state, coupled with a Deep Reinforcement Learning policy (SAC or PPO). By decoupling feature extraction from policy learning, the approach achieves significant reductions in model size and training time while improving information throughput, evidenced by higher probability-efficiency metrics and more stable training than frame-stacking baselines. Empirically, LSTMAE_SAC emerges as the strongest performer, delivering superior rollout rewards, longer effective exploration, and lower parameter counts compared to large-frame-stacking networks, with large gains in training efficiency. The findings underscore the importance of architecture design and decoupled representation learning when applying DRL to high-dimensional, temporally structured observation spaces, particularly in time-sensitive, resource-constrained WiSAR scenarios.

Abstract

Wilderness search and rescue operations are often carried out over vast landscapes. The search efforts, however, must be undertaken in minimum time to maximize the chance of survival of the victim. Whilst the advent of cheap multicopters in recent years has changed the way search operations are handled, it has not solved the challenges of the massive areas at hand. The problem therefore is not one of complete coverage, but one of maximizing the information gathered in the limited time available. In this work we propose that a combination of a recurrent autoencoder and deep reinforcement learning is a more efficient solution to the search problem than previous pure deep reinforcement learning or optimisation approaches. The autoencoder training paradigm efficiently maximizes the information throughput of the encoder into its latent space representation which deep reinforcement learning is primed to leverage. Without the overhead of independently solving the problem that the recurrent autoencoder is designed for, it is more efficient in learning the control task. We further implement three additional architectures for a comprehensive comparison of the main proposed architecture. Similarly, we apply both soft actor-critic and proximal policy optimisation to provide an insight into the performance of both in a highly non-linear and complex application with a large observation Results show that the proposed architecture is vastly superior to the benchmarks, with soft actor-critic achieving the best performance. This model further outperformed work from the literature whilst having below a fifth of the total learnable parameters and training in a quarter of the time.

Recurrent Auto-Encoders for Enhanced Deep Reinforcement Learning in Wilderness Search and Rescue Planning

TL;DR

The paper tackles WiSAR planning over vast terrains where full coverage is impractical by introducing a Recurrent Autoencoder (LSTMAE) that preprocesses temporal observations into a compact latent state, coupled with a Deep Reinforcement Learning policy (SAC or PPO). By decoupling feature extraction from policy learning, the approach achieves significant reductions in model size and training time while improving information throughput, evidenced by higher probability-efficiency metrics and more stable training than frame-stacking baselines. Empirically, LSTMAE_SAC emerges as the strongest performer, delivering superior rollout rewards, longer effective exploration, and lower parameter counts compared to large-frame-stacking networks, with large gains in training efficiency. The findings underscore the importance of architecture design and decoupled representation learning when applying DRL to high-dimensional, temporally structured observation spaces, particularly in time-sensitive, resource-constrained WiSAR scenarios.

Abstract

Wilderness search and rescue operations are often carried out over vast landscapes. The search efforts, however, must be undertaken in minimum time to maximize the chance of survival of the victim. Whilst the advent of cheap multicopters in recent years has changed the way search operations are handled, it has not solved the challenges of the massive areas at hand. The problem therefore is not one of complete coverage, but one of maximizing the information gathered in the limited time available. In this work we propose that a combination of a recurrent autoencoder and deep reinforcement learning is a more efficient solution to the search problem than previous pure deep reinforcement learning or optimisation approaches. The autoencoder training paradigm efficiently maximizes the information throughput of the encoder into its latent space representation which deep reinforcement learning is primed to leverage. Without the overhead of independently solving the problem that the recurrent autoencoder is designed for, it is more efficient in learning the control task. We further implement three additional architectures for a comprehensive comparison of the main proposed architecture. Similarly, we apply both soft actor-critic and proximal policy optimisation to provide an insight into the performance of both in a highly non-linear and complex application with a large observation Results show that the proposed architecture is vastly superior to the benchmarks, with soft actor-critic achieving the best performance. This model further outperformed work from the literature whilst having below a fifth of the total learnable parameters and training in a quarter of the time.

Paper Structure

This paper contains 32 sections, 1 theorem, 15 equations, 7 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Environment Modelling
  4. Agent Dynamics
  5. Probability Distribution Map
  6. Reward Architecture
  7. Observation Processing
  8. Recurrent Autoencoder
  9. Model Architectures
  10. Practical Implementation Details
  11. Cubature
  12. Recurrent Encoder
  13. Training
  14. Deployment
  15. Further Architecture Details
  16. ...and 17 more sections

Key Result

Lemma 3.1

For a uniform PDM, $p(\mathbf{x}) = 1$, the area $A(\theta)$ of the region $H$ after two steps, as defined by where ${R_\textit{buffer}}$ and $s$ are constants, is maximized when $\theta = 0$.

Figures (7)

  • Figure 1: Example PDM $p(\mathbf{x})$ with ${N_\textit{gaussian}} = 3$ and constant covariance. Lighter areas are of higher probability whilst darker ones have lower probability. During search planning the agent would avoid lower probability regions whilst targeting high probability ones to maximize total seen probability.
  • Figure 2: Visualizing $H_2$ ($H_t$ after two steps) with different $\theta$. The areas coloured in red and green represent $f(\theta)$ and $g(\theta)$ from \ref{['eqn:method:area_of_two_steps']} respectively.
  • Figure 3: The Long Short-Term Memory unit internal structure from \ref{['eqn:lstm']}
  • Figure 4: RAE architecture using LSTMs for encoding and decoding. Using an unbalanced architecture, with the decoder being larger than the encoder, enables higher quality reconstruction which results in faster training and better performance. Softsign is applied to the decoder output to ensure that values meet the $s_\mathrm{path} \in [-1,1]$ requirement whilst providing close to linear mappings in this range.
  • Figure 5: The five proposed policy architectures for use with either PPO, RPPO, or SAC. Figure \ref{['fig:method:rppo_policy']} is only used with RPPO.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Lemma 3.1
  • proof