Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TCRL: Temporal-Coupled Adversarial Training for Robust Constrained Reinforcement Learning in Worst-Case Scenarios

Wentao Xu, Zhongming Yao, Weihao Li, Zhenghang Song, Yumeng Song, Tianyi Li, Yushuai Li

TL;DR

TCRL introduces a worst-case-perceived cost constraint function that estimates safety costs under temporally coupled perturbations without the need to explicitly model adversarial attackers, and establishes a dual-constraint defense mechanism on the reward to counter temporally coupled adversaries while maintaining reward unpredictability.

Abstract

Constrained Reinforcement Learning (CRL) aims to optimize decision-making policies under constraint conditions, making it highly applicable to safety-critical domains such as autonomous driving, robotics, and power grid management. However, existing robust CRL approaches predominantly focus on single-step perturbations and temporally independent adversarial models, lacking explicit modeling of robustness against temporally coupled perturbations. To tackle these challenges, we propose TCRL, a novel temporal-coupled adversarial training framework for robust constrained reinforcement learning (TCRL) in worst-case scenarios. First, TCRL introduces a worst-case-perceived cost constraint function that estimates safety costs under temporally coupled perturbations without the need to explicitly model adversarial attackers. Second, TCRL establishes a dual-constraint defense mechanism on the reward to counter temporally coupled adversaries while maintaining reward unpredictability. Experimental results demonstrate that TCRL consistently outperforms existing methods in terms of robustness against temporally coupled perturbation attacks across a variety of CRL tasks.

TCRL: Temporal-Coupled Adversarial Training for Robust Constrained Reinforcement Learning in Worst-Case Scenarios

TL;DR

TCRL introduces a worst-case-perceived cost constraint function that estimates safety costs under temporally coupled perturbations without the need to explicitly model adversarial attackers, and establishes a dual-constraint defense mechanism on the reward to counter temporally coupled adversaries while maintaining reward unpredictability.

Abstract

Constrained Reinforcement Learning (CRL) aims to optimize decision-making policies under constraint conditions, making it highly applicable to safety-critical domains such as autonomous driving, robotics, and power grid management. However, existing robust CRL approaches predominantly focus on single-step perturbations and temporally independent adversarial models, lacking explicit modeling of robustness against temporally coupled perturbations. To tackle these challenges, we propose TCRL, a novel temporal-coupled adversarial training framework for robust constrained reinforcement learning (TCRL) in worst-case scenarios. First, TCRL introduces a worst-case-perceived cost constraint function that estimates safety costs under temporally coupled perturbations without the need to explicitly model adversarial attackers. Second, TCRL establishes a dual-constraint defense mechanism on the reward to counter temporally coupled adversaries while maintaining reward unpredictability. Experimental results demonstrate that TCRL consistently outperforms existing methods in terms of robustness against temporally coupled perturbation attacks across a variety of CRL tasks.
Paper Structure (25 sections, 1 theorem, 20 equations, 1 figure, 3 tables, 1 algorithm)

This paper contains 25 sections, 1 theorem, 20 equations, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminary
  4. CRL.
  5. Modeling State Observation Perturbation Attacks.
  6. Methodology
  7. Framework
  8. Temporal-coupled Perturbation Attack
  9. Cost Constraint Functions
  10. Reward-based Temporal Defense Constraints
  11. Practical Implementation
  12. Experiment
  13. Adversarial Attacker
  14. Experimental Settings
  15. Performance of CRL with Robust Training
  16. ...and 10 more sections

Key Result

theorem 1

For any strategy $\pi$, the operator $\Gamma$ is a compression mapping whose unique immovable point ${Q}^{\pi}_c$ is the worst-cost value of the strategy under an attack on the bounded observation of the $\ell _p$-paradigm, where the upper bound on the attack radius is $\epsilon$.

Figures (1)

  • Figure 1: TCRL's framework.

Theorems & Definitions (5)

  • Definition 1: Temporal-coupled Perturbation Constraint Parameter
  • Definition 2: $\bar{\epsilon}_{t}$-Temporal-coupled State Perturbation
  • Definition 3: Worst-case Cost Bellman Operator
  • theorem 1: Worst-case Cost Bellman Operator with Worst-case Cost Values
  • Definition 4: Worst-case Cost Value