Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training Environment for High Performance Reinforcement Learning

Greg Search

TL;DR

Tunnel addresses the need for rapid evaluation of autonomous high-performance air combat by providing an open-source, Gymnasium-compatible training environment that embeds the F-16 3D nonlinear dynamics. The core approach is a configurable tunnel-like task with customizable observations, actions, sensory models, and mission constructs, enabling quick trade studies across learning methods and sensor configurations. Key contributions include a practical, modifiable environment that supports reinforcement and imitation learning, plus a missionized extension exploring adversaries and GPS-denied operations, demonstrated through a limited trade study. The work facilitates collaboration between researchers and mission planners, potentially accelerating operational relevance and software agility in autonomous air combat.

Abstract

This paper presents Tunnel, a simple, open source, reinforcement learning training environment for high performance aircraft. It integrates the F16 3D nonlinear flight dynamics into OpenAI Gymnasium python package. The template includes primitives for boundaries, targets, adversaries and sensing capabilities that may vary depending on operational need. This offers mission planners a means to rapidly respond to evolving environments, sensor capabilities and adversaries for autonomous air combat aircraft. It offers researchers access to operationally relevant aircraft physics. Tunnel code base is accessible to anyone familiar with Gymnasium and/or those with basic python skills. This paper includes a demonstration of a week long trade study that investigated a variety of training methods, observation spaces, and threat presentations. This enables increased collaboration between researchers and mission planners which can translate to a national military advantage. As warfare becomes increasingly reliant upon automation, software agility will correlate with decision advantages. Airmen must have tools to adapt to adversaries in this context. It may take months for researchers to develop skills to customize observation, actions, tasks and training methodologies in air combat simulators. In Tunnel, this can be done in a matter of days.

Training Environment for High Performance Reinforcement Learning

TL;DR

Tunnel addresses the need for rapid evaluation of autonomous high-performance air combat by providing an open-source, Gymnasium-compatible training environment that embeds the F-16 3D nonlinear dynamics. The core approach is a configurable tunnel-like task with customizable observations, actions, sensory models, and mission constructs, enabling quick trade studies across learning methods and sensor configurations. Key contributions include a practical, modifiable environment that supports reinforcement and imitation learning, plus a missionized extension exploring adversaries and GPS-denied operations, demonstrated through a limited trade study. The work facilitates collaboration between researchers and mission planners, potentially accelerating operational relevance and software agility in autonomous air combat.

Abstract

This paper presents Tunnel, a simple, open source, reinforcement learning training environment for high performance aircraft. It integrates the F16 3D nonlinear flight dynamics into OpenAI Gymnasium python package. The template includes primitives for boundaries, targets, adversaries and sensing capabilities that may vary depending on operational need. This offers mission planners a means to rapidly respond to evolving environments, sensor capabilities and adversaries for autonomous air combat aircraft. It offers researchers access to operationally relevant aircraft physics. Tunnel code base is accessible to anyone familiar with Gymnasium and/or those with basic python skills. This paper includes a demonstration of a week long trade study that investigated a variety of training methods, observation spaces, and threat presentations. This enables increased collaboration between researchers and mission planners which can translate to a national military advantage. As warfare becomes increasingly reliant upon automation, software agility will correlate with decision advantages. Airmen must have tools to adapt to adversaries in this context. It may take months for researchers to develop skills to customize observation, actions, tasks and training methodologies in air combat simulators. In Tunnel, this can be done in a matter of days.
Paper Structure (25 sections, 6 equations, 5 figures, 1 table)

This paper contains 25 sections, 6 equations, 5 figures, 1 table.

Table of Contents

  1. Nomenclature
  2. Introduction
  3. Related Work
  4. Background
  5. Air Combat Autonomy
  6. F-16 Dynamics
  7. Algorithms and Training Methods
  8. Sensing in Contested Environments
  9. Simulations vs Training Environments
  10. Design
  11. Features
  12. Description
  13. Trade Study
  14. Reinforcement Learning
  15. Custom Observations & Actions
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Human and Agent Executed Air Combat
  • Figure 2: Axes Conventions, credit Heidlauf et al. verification
  • Figure 3: Tunnel Training Environment
  • Figure 4: Imagery Input: LiDAR returns at each node colored by distance
  • Figure 5: Missionized Tunnel Environment