Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Use of Reinforcement Learning for Attacking and Defending Load Frequency Control

Amr S. Mohamed, Deepa Kundur

TL;DR

This work addresses vulnerabilities in load frequency control (LFC) by applying deep reinforcement learning to synthesize false data injection and load-switching attacks, producing diverse attack datasets on microgrid models. The authors develop an RL-based attacker using deep deterministic policy gradient (DDPG) and leverage RL-generated data to train a supervised LSTM detector and compare it with a state-of-the-art unsupervised autoencoder detector, highlighting strengths and weaknesses of each approach. They also propose an integrated defense that combines anomaly detection with supervised attack detection, achieving high accuracy in identifying malicious activity while reducing false positives. The study demonstrates how RL-based attack synthesis can inform vulnerability testing and defense design, enabling proactive security for cyber-physical power systems despite limitations such as training time and model simplifications.

Abstract

The electric grid is an attractive target for cyberattackers given its critical nature in society. With the increasing sophistication of cyberattacks, effective grid defense will benefit from proactively identifying vulnerabilities and attack strategies. We develop a deep reinforcement learning-based method that recognizes vulnerabilities in load frequency control, an essential process that maintains grid security and reliability. We demonstrate how our method can synthesize a variety of attacks involving false data injection and load switching, while specifying the attack and threat models - providing insight into potential attack strategies and impact. We discuss how our approach can be employed for testing electric grid vulnerabilities. Moreover our method can be employed to generate data to inform the design of defense strategies and develop attack detection methods. For this, we design and compare a (deep learning-based) supervised attack detector with an unsupervised anomaly detector to highlight the benefits of developing defense strategies based on identified attack strategies.

On the Use of Reinforcement Learning for Attacking and Defending Load Frequency Control

TL;DR

This work addresses vulnerabilities in load frequency control (LFC) by applying deep reinforcement learning to synthesize false data injection and load-switching attacks, producing diverse attack datasets on microgrid models. The authors develop an RL-based attacker using deep deterministic policy gradient (DDPG) and leverage RL-generated data to train a supervised LSTM detector and compare it with a state-of-the-art unsupervised autoencoder detector, highlighting strengths and weaknesses of each approach. They also propose an integrated defense that combines anomaly detection with supervised attack detection, achieving high accuracy in identifying malicious activity while reducing false positives. The study demonstrates how RL-based attack synthesis can inform vulnerability testing and defense design, enabling proactive security for cyber-physical power systems despite limitations such as training time and model simplifications.

Abstract

The electric grid is an attractive target for cyberattackers given its critical nature in society. With the increasing sophistication of cyberattacks, effective grid defense will benefit from proactively identifying vulnerabilities and attack strategies. We develop a deep reinforcement learning-based method that recognizes vulnerabilities in load frequency control, an essential process that maintains grid security and reliability. We demonstrate how our method can synthesize a variety of attacks involving false data injection and load switching, while specifying the attack and threat models - providing insight into potential attack strategies and impact. We discuss how our approach can be employed for testing electric grid vulnerabilities. Moreover our method can be employed to generate data to inform the design of defense strategies and develop attack detection methods. For this, we design and compare a (deep learning-based) supervised attack detector with an unsupervised anomaly detector to highlight the benefits of developing defense strategies based on identified attack strategies.
Paper Structure (23 sections, 19 equations, 13 figures, 4 tables, 1 algorithm)

This paper contains 23 sections, 19 equations, 13 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Frequency Control and Protection
  4. Reinforcement Learning
  5. Threat Model
  6. Method
  7. System Architecture
  8. RL for Attacking Load Frequency Control
  9. Unsupervised Autoencoder-based Anomaly Detection
  10. Supervised attack detection using RL generated data
  11. Results
  12. False data injection
  13. Load switching
  14. Supervised attack detection
  15. Unsupervised anomaly detection
  16. ...and 8 more sections

Figures (13)

  • Figure 1: Change in load power as a stochastic process with $\sigma_1 = (0.05/3)$ and $\sigma_2 = (0.2/3)$ to simulate normal grid operation data.
  • Figure 2: Microgrid testbed. SG is the synchronous generator which is target for the attacks.
  • Figure 3: False data injection attacks reward function. The agent is rewarded for increasing the rate of change of frequency while maintaining small frequency deviation.
  • Figure 4: Detailed model simulations: Demonstrating the ability of the RL agent to attack different systems. Each column corresponds to a different system -- with eigenmodes located at (left: MG2) 4 and (right: MG3) 3.4 rad/s. The RL agent is able to adapt its attack.
  • Figure 5: Load switching attack reward function. The agent is rewarded for increasing the frequency deviation and its rate of change towards the protection relay settings in red.
  • ...and 8 more figures