BVR Gym: A Reinforcement Learning Environment for Beyond-Visual-Range Air Combat

TL;DR

The paper addresses the need for an open, high-fidelity environment to study beyond-visual-range air combat using reinforcement learning. It introduces BVR Gym, a JSBSim-based, open-source platform with an F-16 aircraft model and a PN-guided BVR missile, augmented by Behavior Trees for manual policies and an OpenAI Gym–like interface to support PPO and other RL algorithms. Three scenarios—evading a single missile, evading two missiles, and a BVR dogfight—demonstrate the framework's ability to probe different tactical challenges and policy structures. The results show that PPO-trained agents can learn effective evasive and combat strategies, with practical guidance on reducing compute and tuning parameters, highlighting BVR Gym’s potential to accelerate research and development of BVR tactics and defense applications.

Abstract

Creating new air combat tactics and discovering novel maneuvers can require numerous hours of expert pilots' time. Additionally, for each different combat scenario, the same strategies may not work since small changes in equipment performance may drastically change the air combat outcome. For this reason, we created a reinforcement learning environment to help investigate potential air combat tactics in the field of beyond-visual-range (BVR) air combat: the BVR Gym. This type of air combat is important since long-range missiles are often the first weapon to be used in aerial combat. Some existing environments provide high-fidelity simulations but are either not open source or are not adapted to the BVR air combat domain. Other environments are open source but use less accurate simulation models. Our work provides a high-fidelity environment based on the open-source flight dynamics simulator JSBSim and is adapted to the BVR air combat domain. This article describes the building blocks of the environment and some use cases.

Figures (8)

• Figure 1: BT structure controlling the F16 aircraft belonging to Team $\mathcal{R}$ .
• Figure 2: F-16 evasive maneuver. A drop in altitude initiates the maneuver, followed by a turn to evade the incoming missile.
• Figure 3: Missile response to F-16 evasive maneuver. Initialized by acceleration stage to Mach 4 and an ascent in altitude.
• Figure 4: Missile response to F-16 evasive maneuver. Initialized by acceleration stage to Mach 4 and an ascent in altitude.
• Figure 5: Training results for evading one and two missile scenarios. The Y-axis shows the average distance by which the missile misses its target in kilometers.