Simulation-based Scenario Generation for Robust Hybrid AI for Autonomy

TL;DR

This work provides a simulation-based autonomy software framework that supports the training, testing and assurance of neuro-symbolic algorithms for autonomous maneuver and perception reasoning and includes scenario generation capabilities that offer mission-relevant contextual symbolic information in addition to raw sensor data.

Abstract

Application of Unmanned Aerial Vehicles (UAVs) in search and rescue, emergency management, and law enforcement has gained traction with the advent of low-cost platforms and sensor payloads. The emergence of hybrid neural and symbolic AI approaches for complex reasoning is expected to further push the boundaries of these applications with decreasing levels of human intervention. However, current UAV simulation environments lack semantic context suited to this hybrid approach. To address this gap, HAMERITT (Hybrid Ai Mission Environment for RapId Training and Testing) provides a simulation-based autonomy software framework that supports the training, testing and assurance of neuro-symbolic algorithms for autonomous maneuver and perception reasoning. HAMERITT includes scenario generation capabilities that offer mission-relevant contextual symbolic information in addition to raw sensor data. Scenarios include symbolic descriptions for entities of interest and their relations to scene elements, as well as spatial-temporal constraints in the form of time-bounded areas of interest with prior probabilities and restricted zones within those areas. HAMERITT also features support for training distinct algorithm threads for maneuver vs. perception within an end-to-end mission run. Future work includes improving scenario realism and scaling symbolic context generation through automated workflow.

Figures (5)

• Figure 1: The HAMERITT architecture includes 1) Scenario & Data Generation with tools for definition, composition, and randomization, 2) a high-fidelity Microsoft UAV simulator, and 3) an Algorithm Development Kit (ADK) with a baseline autonomy stack and COP-building pipeline with APIs tailored to ANSR performer roles.
• Figure 2: HAMERITT supports Area Search (top) and Route Search (bottom) scenario classes, with contextual symbolic information for spatial-temporal constraints and prior entity location belief maps.
• Figure 3: HAMERITT includes vignettes to support reasoning about complex events. The vignette showing two pedestrians chasing the red car provide an instance of anomalous scene. The subsequent trajectory followed by the red car enables a pursuit challenge for a UAV.
• Figure 4: A "RIGHT_OF" spatial relationship between target car (Car_123) and a garage.
• Figure 5: HAMERITT uses a realistic Microsoft UAV simulator to dynamically spawn entities of interest (top) and simulate realistic environmental variations including challenging weather and visibility conditions (bottom).