Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating False Alarm and Missing Attacks in CAN IDS

Nirab Hossain, Pablo Moriano

TL;DR

This paper evaluates the adversarial robustness of machine learning based CAN intrusion detection systems (IDS) using the ROAD dataset. It compares shallow architectures (DT, RF, ET, XGB) and a deep neural network under protocol-constrained perturbations generated by FGSM, BIM, and PGD, focusing on both inducing false alarms and causing missed detections. Results show that while benign performance remains strong, gradient-based attacks can severely impair detection, with the deep model being particularly vulnerable to missed-attacks in several scenarios; the ET ensemble offers the best overall robustness. The study highlights the necessity of adversarial robustness assessments in safety-critical automotive IDS and discusses defense directions such as adversarial training and hybrid designs for future CV/ICS security in vehicles.

Abstract

Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern. Machine learning (ML)-based intrusion detection systems (IDS) are increasingly deployed to protect CAN traffic, yet their robustness against adversarial manipulation remains largely unexplored. We present a systematic adversarial evaluation of CAN IDS using the ROAD dataset, comparing four shallow learning models with a deep neural network-based detector. Using protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD, we evaluate adversarial effects on both benign and malicious CAN frames. While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities. Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks. Notably, under gradient-based attacks, the shallow model extra trees (ET) demonstrates improved robustness to missed-attack induction compared to the other models. Our results demonstrate that adversarial manipulation can simultaneously trigger false alarms and evade detection, underscoring the need for adversarial robustness evaluation in safety-critical automotive IDS.

Evaluating False Alarm and Missing Attacks in CAN IDS

TL;DR

This paper evaluates the adversarial robustness of machine learning based CAN intrusion detection systems (IDS) using the ROAD dataset. It compares shallow architectures (DT, RF, ET, XGB) and a deep neural network under protocol-constrained perturbations generated by FGSM, BIM, and PGD, focusing on both inducing false alarms and causing missed detections. Results show that while benign performance remains strong, gradient-based attacks can severely impair detection, with the deep model being particularly vulnerable to missed-attacks in several scenarios; the ET ensemble offers the best overall robustness. The study highlights the necessity of adversarial robustness assessments in safety-critical automotive IDS and discusses defense directions such as adversarial training and hybrid designs for future CV/ICS security in vehicles.

Abstract

Modern vehicles rely on electronic control units (ECUs) interconnected through the Controller Area Network (CAN), making in-vehicle communication a critical security concern. Machine learning (ML)-based intrusion detection systems (IDS) are increasingly deployed to protect CAN traffic, yet their robustness against adversarial manipulation remains largely unexplored. We present a systematic adversarial evaluation of CAN IDS using the ROAD dataset, comparing four shallow learning models with a deep neural network-based detector. Using protocol-compliant, payload-level perturbations generated via FGSM, BIM and PGD, we evaluate adversarial effects on both benign and malicious CAN frames. While all models achieve strong baseline performance under benign conditions, adversarial perturbations reveal substantial vulnerabilities. Although shallow and deep models are robust to false-alarm induction, with the deep neural network (DNN) performing best on benign traffic, all architectures suffer significant increases in missed attacks. Notably, under gradient-based attacks, the shallow model extra trees (ET) demonstrates improved robustness to missed-attack induction compared to the other models. Our results demonstrate that adversarial manipulation can simultaneously trigger false alarms and evade detection, underscoring the need for adversarial robustness evaluation in safety-critical automotive IDS.
Paper Structure (18 sections, 2 equations, 2 figures, 8 tables)

This paper contains 18 sections, 2 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Materials and Methods
  4. CAN
  5. IDS Architecture
  6. Threat Model
  7. Dataset
  8. Preprocessing
  9. Domain Constraints
  10. Adversarial Attack Methods
  11. Evaluation Metrics
  12. Results
  13. Overview of Evaluation
  14. Benign Performance and False-Alarm Robustness
  15. Adversarial Performance and Missed-Attack Vulnerability
  16. ...and 3 more sections

Figures (2)

  • Figure 1: CAN data frame structure.
  • Figure 2: Workflow for IDS training, adversarial IVN frame generation, and evaluation of benign and adversarial predictions with FN, FP, and MCC.