A Domain-Knowledge-Aided Deep Reinforcement Learning Approach for Flight Control Design

TL;DR

The proposed approach fixes the autopilot structure as typical three-loop autopilot and deep reinforcement learning is utilised to learn the autopilot gains and formulates a Markovian decision process with a proper reward function that enable the application of reinforcement learning theory.

Abstract

This paper aims to examine the potential of using the emerging deep reinforcement learning techniques in flight control. Instead of learning from scratch, we suggest to leverage domain knowledge available in learning to improve learning efficiency and generalisability. More specifically, the proposed approach fixes the autopilot structure as typical three-loop autopilot and deep reinforcement learning is utilised to learn the autopilot gains. To solve the flight control problem, we then formulate a Markovian decision process with a proper reward function that enable the application of reinforcement learning theory. Another type of domain knowledge is exploited for defining the reward function, by shaping reference inputs in consideration of important control objectives and using the shaped reference inputs in the reward function. The state-of-the-art deep deterministic policy gradient algorithm is utilised to learn an action policy that maps the observed states to the autopilot gains. Extensive empirical numerical simulations are performed to validate the proposed computational control algorithm.

Figures (15)

• Figure 1: Basic concept of DDPG.
• Figure 2: The longitudinal dynamics model and parameter definitions.
• Figure 3: Three-loop autopilot structure.
• Figure 4: Information flow of the proposed RL framework.
• Figure 5: Comparisons of learning curves.