Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Collaborative Autonomy in Opposed Environments using Maritime Capture-the-Flag Competitions

Jordan Beason, Michael Novitzky, John Kliem, Tyler Errico, Zachary Serlin, Kevin Becker, Tyler Paine, Michael Benjamin, Prithviraj Dasgupta, Peter Crowley, Charles O'Donnell, John James

TL;DR

The results of the experiment demonstrate that rule-based cooperation for behavior-based agents outperformed those trained in Deep-reinforcement learning paradigms as implemented in these competitions, demonstrating that the competitive gap between behavior-based autonomy and deep RL will be reduced.

Abstract

The objective of this work is to evaluate multi-agent artificial intelligence methods when deployed on teams of unmanned surface vehicles (USV) in an adversarial environment. Autonomous agents were evaluated in real-world scenarios using the Aquaticus test-bed, which is a Capture-the-Flag (CTF) style competition involving teams of USV systems. Cooperative teaming algorithms of various foundations in behavior-based optimization and deep reinforcement learning (RL) were deployed on these USV systems in two versus two teams and tested against each other during a competition period in the fall of 2023. Deep reinforcement learning applied to USV agents was achieved via the Pyquaticus test bed, a lightweight gymnasium environment that allows simulated CTF training in a low-level environment. The results of the experiment demonstrate that rule-based cooperation for behavior-based agents outperformed those trained in Deep-reinforcement learning paradigms as implemented in these competitions. Further integration of the Pyquaticus gymnasium environment for RL with MOOS-IvP in terms of configuration and control schema will allow for more competitive CTF games in future studies. As the development of experimental deep RL methods continues, the authors expect that the competitive gap between behavior-based autonomy and deep RL will be reduced. As such, this report outlines the overall competition, methods, and results with an emphasis on future works such as reward shaping and sim-to-real methodologies and extending rule-based cooperation among agents to react to safety and security events in accordance with human experts intent/rules for executing safety and security processes.

Evaluating Collaborative Autonomy in Opposed Environments using Maritime Capture-the-Flag Competitions

TL;DR

The results of the experiment demonstrate that rule-based cooperation for behavior-based agents outperformed those trained in Deep-reinforcement learning paradigms as implemented in these competitions, demonstrating that the competitive gap between behavior-based autonomy and deep RL will be reduced.

Abstract

The objective of this work is to evaluate multi-agent artificial intelligence methods when deployed on teams of unmanned surface vehicles (USV) in an adversarial environment. Autonomous agents were evaluated in real-world scenarios using the Aquaticus test-bed, which is a Capture-the-Flag (CTF) style competition involving teams of USV systems. Cooperative teaming algorithms of various foundations in behavior-based optimization and deep reinforcement learning (RL) were deployed on these USV systems in two versus two teams and tested against each other during a competition period in the fall of 2023. Deep reinforcement learning applied to USV agents was achieved via the Pyquaticus test bed, a lightweight gymnasium environment that allows simulated CTF training in a low-level environment. The results of the experiment demonstrate that rule-based cooperation for behavior-based agents outperformed those trained in Deep-reinforcement learning paradigms as implemented in these competitions. Further integration of the Pyquaticus gymnasium environment for RL with MOOS-IvP in terms of configuration and control schema will allow for more competitive CTF games in future studies. As the development of experimental deep RL methods continues, the authors expect that the competitive gap between behavior-based autonomy and deep RL will be reduced. As such, this report outlines the overall competition, methods, and results with an emphasis on future works such as reward shaping and sim-to-real methodologies and extending rule-based cooperation among agents to react to safety and security events in accordance with human experts intent/rules for executing safety and security processes.
Paper Structure (19 sections, 1 equation, 7 figures, 2 tables)

This paper contains 19 sections, 1 equation, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. The Project Aquaticus Test-bed
  4. Game Setup and Rules
  5. An Introduction to MOOS-IvP
  6. Entries
  7. Behavior Based Autonomy using MOOS-IvP
  8. Pav01 - The Default Aquaticus Opponent
  9. Static Roles and Rule-based Switching
  10. Rule-based strategy Using Opponent Observation and Classification
  11. Pyquaticus Environment
  12. Formal Game Representation
  13. Pyquaticus Reward Functions
  14. Deep Reinforcement Learning using Pyquaticus
  15. Results and Implications from October 2023 Experiments
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Sea Robotics Surveyor Class USV's from field experiments in October 2023.
  • Figure 2: Above: Playback of an Aquaticus CTF game, red two is tagged and returning to base, while blue two is returning to base with a "grabbed" flag. Below: The ocean gameplay environment
  • Figure 3: A visualization of the IvP Helm arcitecture.
  • Figure 4: An Example of the default Pav01 behavior, one attacker boat and a loiter behavior defense agent.
  • Figure 5: Mode tree for Behavior-Based Strategies. The OpRegion and AvoidCollision behaviors are not shown, but were always active.
  • ...and 2 more figures