ARM-IRL: Adaptive Resilience Metric Quantification Using Inverse Reinforcement Learning

TL;DR

The paper tackles adaptive resilience quantification for cyber-physical power systems by learning a time- and state-dependent resilience metric $R_{adapt}(t,s)$ from expert demonstrations using adversarial inverse reinforcement learning (AIRL). It formulates three MDPs—rerouting, distribution feeder reconfiguration, and combined cyber-physical restoration—to evaluate adaptive resilience learning and compares imitation-learning variants (BC, DAgger, GAIL, AIRL). AIRL consistently offers superior sample efficiency and robustness, achieving faster goal attainment and more reliable reward signals than alternatives. The proposed ARM-IRL architecture integrates OpenDSS and SimPy with a static resilience prior to learn both the metric and the policy, showing promise for scalable resilience management in cyber-physical grids and guiding future extensions to larger networks and multi-agent scenarios.

Abstract

Resilience of safety-critical systems is gaining importance, particularly with the increasing number of cyber and physical threats. Cyber-physical threats are becoming increasingly prevalent, as digital systems are ubiquitous in critical infrastructure. The challenge with determining the resilience of cyber-physical systems is identifying a set of resilience metrics that can adapt to the changing states of the system. A static resilience metric can lead to an inaccurate estimation of system state, and can result in unintended consequences against cyber threats. In this work, we propose a data-driven method for adaptive resilience metric learning. The primary goal is to learn a single resilience metric by formulating an inverse reinforcement learning problem that learns a reward or objective from a set of control actions from an expert. It learns the structure or parameters of the reward function based on information provided by expert demonstrations. Most prior work has considered static weights or theories from fuzzy logic to formulate a single resilience metric. Instead, this work learns the resilience metric, represented as reward function, using adversarial inverse reinforcement learning, to determine the optimal policy through training the generator discriminator in parallel. We evaluate our proposed technique in scenarios such as optimal communication network rerouting, power distribution network reconfiguration, and a combined cyber-physical restoration of critical load using the IEEE 123-bus system.

Figures (15)

• Figure 1: GAN architecture for policy and reward learning.
• Figure 2: Overall architecture of ARM-IRL.
• Figure 3: IEEE 123 test system segregated to two zones.
• Figure 4: Communication network, $N_6$, where all the six routers are controllable and DoS attack is performed at any of the routers = $R_3,R_4,R_5$
• Figure 5: Communication network, $N_8$, where three out of eight routers are controllable