Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Designing an Objective-Driven Test Method for the Comparative Performance Evaluation of Commercial DTI Solutions for Counter UAS systems

Ali Mohamoud, Johan van de Pol, Hanno Hildmann, Rob van Heijster, Beatrice Masini, Martijn van den Heuvel, Amber van Keeken

TL;DR

This work tackles the lack of objective benchmarks for Counter-UAS DTI systems by proposing an objective-driven, scenario-based testing framework capable of fair, end-user–oriented benchmarking. It integrates a configurable objective function with a modular evaluation pipeline that aggregates metric-level results into component and system-level scores, grounded in real-world operational needs. The methodology is validated through both simulation-based and trial-based approaches, demonstrating the ability to evaluate detection, tracking, and identification under varied environmental conditions and across multiple vendors. The framework is designed to be future-proof and adaptable, supporting transparent decision-making for procurement and standardization efforts in C-UAS DTI deployments.

Abstract

Unmanned Aerial Systems (UASs) or drones become more and more commercially available and cheap. There has been much emphasis on developing and deploying Counter-UAS systems (UASs) with Detection Tracking and Identification (DTI) solutions. However, the capabilities of these systems are hard to benchmark. Performance claims of these systems are currently not supported by evidence. In addition, no standard test methodologies are available for these DTI systems and different test methodologies make comparison of these systems hard or impossible. We report on the definition, development and verification of an objective-driven test method and corresponding comparative performance evaluation for commercial DTI solutions for C-UASs. The developed methodology is based on end-user scenarios that are operationally relevant. The test methodology is based on a generic DTI system lay-out and is detailed towards detection, tracking and identification, taking into account contextual information and end-user input. The comparative performance evaluation is developed to enable the use of the methodology in a relevant environment, thereby taking into account any potential environmental aspect that might influence DTI system performance. Validation of the work in a relevant environment has been done in three operational trials. The operational trial results show that the method allows for performance evaluation at component level (i.e., detection, tracking or identification component) and at system level (combinations of these components and integrated DTI system of system solutions).

Designing an Objective-Driven Test Method for the Comparative Performance Evaluation of Commercial DTI Solutions for Counter UAS systems

TL;DR

This work tackles the lack of objective benchmarks for Counter-UAS DTI systems by proposing an objective-driven, scenario-based testing framework capable of fair, end-user–oriented benchmarking. It integrates a configurable objective function with a modular evaluation pipeline that aggregates metric-level results into component and system-level scores, grounded in real-world operational needs. The methodology is validated through both simulation-based and trial-based approaches, demonstrating the ability to evaluate detection, tracking, and identification under varied environmental conditions and across multiple vendors. The framework is designed to be future-proof and adaptable, supporting transparent decision-making for procurement and standardization efforts in C-UAS DTI deployments.

Abstract

Unmanned Aerial Systems (UASs) or drones become more and more commercially available and cheap. There has been much emphasis on developing and deploying Counter-UAS systems (UASs) with Detection Tracking and Identification (DTI) solutions. However, the capabilities of these systems are hard to benchmark. Performance claims of these systems are currently not supported by evidence. In addition, no standard test methodologies are available for these DTI systems and different test methodologies make comparison of these systems hard or impossible. We report on the definition, development and verification of an objective-driven test method and corresponding comparative performance evaluation for commercial DTI solutions for C-UASs. The developed methodology is based on end-user scenarios that are operationally relevant. The test methodology is based on a generic DTI system lay-out and is detailed towards detection, tracking and identification, taking into account contextual information and end-user input. The comparative performance evaluation is developed to enable the use of the methodology in a relevant environment, thereby taking into account any potential environmental aspect that might influence DTI system performance. Validation of the work in a relevant environment has been done in three operational trials. The operational trial results show that the method allows for performance evaluation at component level (i.e., detection, tracking or identification component) and at system level (combinations of these components and integrated DTI system of system solutions).
Paper Structure (62 sections, 46 equations, 13 figures, 6 tables)

This paper contains 62 sections, 46 equations, 13 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Scope
  4. Contributions of this paper
  5. Paper Structure
  6. Objective driven and comparative performance testing
  7. Introduction
  8. Objective driven test methods
  9. Comparative performance evaluation
  10. Objective driven, comparative performance testing
  11. Important process stages
  12. System Design Principles
  13. Defining the objective
  14. Scenario modelling
  15. A methodology to evaluate DTI solutions for counter-UAS systems
  16. ...and 47 more sections

Figures (13)

  • Figure S1: A visual guide of the paper, outlining the relation of the sections to one another.
  • Figure S2: Important process steps for comparative performance evaluation.
  • Figure S3: A simplified functional overview over a system. See Sections \ref{['subsec:DTI:Detection']}, \ref{['subsec:DTI:Tracking']} and \ref{['subsec:DTI:Identification']} for more details on these functional components of a system, respectively.
  • Figure S4: Performance evaluation at Metric level. Note that the performance evaluation is using the three metrics for detection, cf. Table \ref{['table.metric.detection']}.
  • Figure S5: Performance evaluation at component level.
  • ...and 8 more figures