Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Cooperative Flight Control Using Visual-Attention

Lianhao Yin, Makram Chahine, Tsun-Hsuan Wang, Tim Seyde, Chao Liu, Mathias Lechner, Ramin Hasani, Daniela Rus

TL;DR

This vision-based air-guardian system combines a causal continuous-depth neural network model with a cooperation layer to enable parallel autonomy between a pilot and a control system based on perceived differences in their attention profiles to balance the trade-off between its level of involvement in the flight and the pilot's expertise and attention.

Abstract

The cooperation of a human pilot with an autonomous agent during flight control realizes parallel autonomy. We propose an air-guardian system that facilitates cooperation between a pilot with eye tracking and a parallel end-to-end neural control system. Our vision-based air-guardian system combines a causal continuous-depth neural network model with a cooperation layer to enable parallel autonomy between a pilot and a control system based on perceived differences in their attention profiles. The attention profiles for neural networks are obtained by computing the networks' saliency maps (feature importance) through the VisualBackProp algorithm, while the attention profiles for humans are either obtained by eye tracking of human pilots or saliency maps of networks trained to imitate human pilots. When the attention profile of the pilot and guardian agents align, the pilot makes control decisions. Otherwise, the air-guardian makes interventions and takes over the control of the aircraft. We show that our attention-based air-guardian system can balance the trade-off between its level of involvement in the flight and the pilot's expertise and attention. The guardian system is particularly effective in situations where the pilot was distracted due to information overload. We demonstrate the effectiveness of our method for navigating flight scenarios in simulation with a fixed-wing aircraft and on hardware with a quadrotor platform.

Towards Cooperative Flight Control Using Visual-Attention

TL;DR

This vision-based air-guardian system combines a causal continuous-depth neural network model with a cooperation layer to enable parallel autonomy between a pilot and a control system based on perceived differences in their attention profiles to balance the trade-off between its level of involvement in the flight and the pilot's expertise and attention.

Abstract

The cooperation of a human pilot with an autonomous agent during flight control realizes parallel autonomy. We propose an air-guardian system that facilitates cooperation between a pilot with eye tracking and a parallel end-to-end neural control system. Our vision-based air-guardian system combines a causal continuous-depth neural network model with a cooperation layer to enable parallel autonomy between a pilot and a control system based on perceived differences in their attention profiles. The attention profiles for neural networks are obtained by computing the networks' saliency maps (feature importance) through the VisualBackProp algorithm, while the attention profiles for humans are either obtained by eye tracking of human pilots or saliency maps of networks trained to imitate human pilots. When the attention profile of the pilot and guardian agents align, the pilot makes control decisions. Otherwise, the air-guardian makes interventions and takes over the control of the aircraft. We show that our attention-based air-guardian system can balance the trade-off between its level of involvement in the flight and the pilot's expertise and attention. The guardian system is particularly effective in situations where the pilot was distracted due to information overload. We demonstrate the effectiveness of our method for navigating flight scenarios in simulation with a fixed-wing aircraft and on hardware with a quadrotor platform.
Paper Structure (17 sections, 2 theorems, 16 equations, 8 figures, 3 tables)

This paper contains 17 sections, 2 theorems, 16 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Setup and Methodology
  4. The Guardian agent's neural model
  5. Visual attention map
  6. Cooperative network
  7. Human-in-the-loop Experiments with a simulated aircraft
  8. Training setup
  9. Results: guardian intervention
  10. Analysis: robustness of the visual attention maps
  11. Analysis: interpretability
  12. Human-in-the-loop results on a real drone
  13. Experimental settings
  14. Neural network structure and training
  15. Experimental results
  16. ...and 2 more sections

Key Result

Theorem 1

Based on Assumptions assumption: saftey set assumption f_ai -assumption: saftey of u = u_ai + delta u, the control policy from Eq. eq: attention switch distance-eq:MPC continous control drives the system into the safety set, $y \in \mathbb{S}$.

Figures (8)

  • Figure 1: Cooperative control of a human pilot and our air-guardian. The air-guardian is an autonomous decision system that cooperates with the pilot to increase the safety of aircraft. A neural network takes the visual observation from the aircraft and predicts the control input $u_{G}$, while the pilot makes a decision and acts with $u_{R}$. A cooperative layer generates the applied control input $u$ using both $u_{G}$ and $u_{R}$ according to perceived attention mismatch.
  • Figure 2: The experimental setup with the eye tracking devices to get the attention profile of the pilot
  • Figure 3: The intervention process during a flight
  • Figure 4: Comparison of a human pilot with a guardian and human pilot
  • Figure 5: Comparison of attention maps (computed by VisualBackProp) between CfC and LSTM. The small dot is light blue and less visible and distinguishable from the rest of the figure (Fig. \ref{['fig: VBP LSTM']}). The small red dot is visible and distinguishable from the rest of the figure (Fig. \ref{['fig: VBP LSTM']}).
  • ...and 3 more figures

Theorems & Definitions (5)

  • Remark 1
  • Theorem 1
  • proof
  • Theorem 2
  • proof