Benchmarking Frameworks and Comparative Studies of Controller Area Network (CAN) Intrusion Detection Systems: A Review

TL;DR

The paper addresses the fragmentation in evaluating CAN intrusion detection systems by proposing a CAN IDS evaluation design space spanning IDS type, attack model, evaluation type, workload, and evaluation metrics. It surveys existing benchmarking frameworks and comparative studies, revealing a strong bias toward anomaly-based methods, a predominance of offline evaluations, and a reliance on real-vehicle datasets with limited coverage of suspension and masquerade attacks. The authors recommend comprehensive benchmarking datasets, online evaluation methods, inclusive Layer 1 and Layer 2 benchmarking, and a richer, standardized set of metrics to enable fair comparisons and practical deployment guidance. This work aims to standardize CAN IDS evaluation, improve reproducibility, and support the selection of robust IDS solutions for automotive networks.

Abstract

The development of intrusion detection systems (IDS) for the in-vehicle Controller Area Network (CAN) bus is one of the main efforts being taken to secure the in-vehicle network against various cyberattacks, which have the potential to cause vehicles to malfunction and result in dangerous accidents. These CAN IDS are evaluated in disparate experimental conditions that vary in terms of the workload used, the features used, the metrics reported, etc., which makes direct comparison difficult. Therefore, there have been several benchmarking frameworks and comparative studies designed to evaluate CAN IDS in similar experimental conditions to understand their relative performance and facilitate the selection of the best CAN IDS for implementation in automotive networks. This work provides a comprehensive survey of CAN IDS benchmarking frameworks and comparative studies in the current literature. A CAN IDS evaluation design space is also proposed in this work, which draws from the wider CAN IDS literature. This is not only expected to serve as a guide for designing CAN IDS evaluation experiments but is also used for categorizing current benchmarking efforts. The surveyed works have been discussed on the basis of the five aspects in the design space-namely IDS type, attack model, evaluation type, workload generation, and evaluation metrics-and recommendations for future work have been identified.

Figures (2)

• Figure 1: CAN IDS Evaluation Design Space
• Figure 2: CAN IDS Workload Types