Analyzing Errors in Controlled Turret System Given Target Location Input from Artificial Intelligence Methods in Automatic Target Recognition

TL;DR

This paper assesses the performance of a controlled gun turret system given target location from an object detector developed from AI methods and defines a measure of object detector error and examines the correlations with several standard metrics in object detection.

Abstract

In this paper, we assess the movement error of a targeting system given target location data from artificial intelligence (AI) methods in automatic target recognition (ATR) systems. Few studies evaluate the impacts on the accuracy in moving a targeting system to an aimpoint provided in this manner. To address this knowledge gap, we assess the performance of a controlled gun turret system given target location from an object detector developed from AI methods. In our assessment, we define a measure of object detector error and examine the correlations with several standard metrics in object detection. We then statistically analyze the object detector error data and turret movement error data acquired from controlled targeting simulations, as well as their aggregate error, to examine the impact on turret movement accuracy. Finally, we study the correlations between additional metrics and the probability of a hit. The results indicate that AI technologies are a significant source of error to targeting systems. Moreover, the results suggest that metrics such as the confidence score, intersection-over-union, average precision and average recall are predictors of accuracy against stationary targets with our system parameters.

Figures (8)

• Figure 1: Illustration of the IoU calculation between a detected bounding box $B_p$ and ground truth bounding box $B_{gt}$ with bounding box corners labeled accordingly.
• Figure 2: Plots of precision against recall from the UAV dataset for two different IoU thresholds using the 11-point interpolation method. The IoU threshold is 0.5 in (\ref{['precisionRecall11ptIou50']}) and 0.75 in (\ref{['precisionRecall11ptIou75']}).
• Figure 3: Block diagram of the control system.
• Figure 4: Example output from two targeting simulations. Note, the images are not part of the dataset used to train the object detection model, but are publicly available. In (\ref{['droneImageBtlc']}), targeting starts from the bottom left corner of the image and in (\ref{['droneImageCenter']}), targeting starts from the center of the image.
• Figure 5: Plots of the AI error against (\ref{['rbgvscscorer1000m']}) the confidence score, (\ref{['rbgvsiour1000m']}) the IoU and (\ref{['rbgvsarear1000m']}) the normalized detected box area; the area is scaled by the largest area out of all detected boxes. Plot (\ref{['iouvscscorer1000m']}) shows the IoU against the confidence score. The total number of observations in each plot is $N=955$.