Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Corruption-Robustness in Performative Reinforcement Learning

Vasilis Pollatos, Debmalya Mandal, Goran Radanovic

TL;DR

This work tackles corruption-robust performative reinforcement learning, where a policy not only shapes the data-generating environment but also faces adversarial data contamination. The authors introduce a corruption-robust repeated retraining framework built on a robust OFTRL and a problem-specific robust gradient estimator, proving last-iterate convergence to an approximately stable policy with error scaling as $O(\sqrt{\epsilon})$. They provide finite-sample convergence guarantees, discuss information-theoretic lower bounds under adversarial gradients, and validate the approach on a gridworld testbed showing robustness to data corruption. The results advance reliable performative RL in practical settings where training data may be partially corrupted and distribution shifts arise from deployment.

Abstract

In performative Reinforcement Learning (RL), an agent faces a policy-dependent environment: the reward and transition functions depend on the agent's policy. Prior work on performative RL has studied the convergence of repeated retraining approaches to a performatively stable policy. In the finite sample regime, these approaches repeatedly solve for a saddle point of a convex-concave objective, which estimates the Lagrangian of a regularized version of the reinforcement learning problem. In this paper, we aim to extend such repeated retraining approaches, enabling them to operate under corrupted data. More specifically, we consider Huber's $ε$-contamination model, where an $ε$ fraction of data points is corrupted by arbitrary adversarial noise. We propose a repeated retraining approach based on convex-concave optimization under corrupted gradients and a novel problem-specific robust mean estimator for the gradients. We prove that our approach exhibits last-iterate convergence to an approximately stable policy, with the approximation error linear in $\sqrtε$. We experimentally demonstrate the importance of accounting for corruption in performative RL.

On Corruption-Robustness in Performative Reinforcement Learning

TL;DR

This work tackles corruption-robust performative reinforcement learning, where a policy not only shapes the data-generating environment but also faces adversarial data contamination. The authors introduce a corruption-robust repeated retraining framework built on a robust OFTRL and a problem-specific robust gradient estimator, proving last-iterate convergence to an approximately stable policy with error scaling as . They provide finite-sample convergence guarantees, discuss information-theoretic lower bounds under adversarial gradients, and validate the approach on a gridworld testbed showing robustness to data corruption. The results advance reliable performative RL in practical settings where training data may be partially corrupted and distribution shifts arise from deployment.

Abstract

In performative Reinforcement Learning (RL), an agent faces a policy-dependent environment: the reward and transition functions depend on the agent's policy. Prior work on performative RL has studied the convergence of repeated retraining approaches to a performatively stable policy. In the finite sample regime, these approaches repeatedly solve for a saddle point of a convex-concave objective, which estimates the Lagrangian of a regularized version of the reinforcement learning problem. In this paper, we aim to extend such repeated retraining approaches, enabling them to operate under corrupted data. More specifically, we consider Huber's -contamination model, where an fraction of data points is corrupted by arbitrary adversarial noise. We propose a repeated retraining approach based on convex-concave optimization under corrupted gradients and a novel problem-specific robust mean estimator for the gradients. We prove that our approach exhibits last-iterate convergence to an approximately stable policy, with the approximation error linear in . We experimentally demonstrate the importance of accounting for corruption in performative RL.
Paper Structure (42 sections, 18 theorems, 114 equations, 1 figure, 5 algorithms)

This paper contains 42 sections, 18 theorems, 114 equations, 1 figure, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Convex-concave optimization
  5. Contamination Model
  6. Smooth Convex-Concave Objectives
  7. Analysis of Robust OFTRL
  8. Formal Setting
  9. Policy-dependent Markov Decision Process
  10. Performatively Stable Policy
  11. Occupancy Measures
  12. Data Generation Process
  13. Contamination Model
  14. Corruption-Robust Performative RL
  15. Robust Repeated Retraining
  16. ...and 27 more sections

Key Result

Theorem 1

The output $(\bar{x},\bar{y})$ of Algorithm alg:aoftrl satisfies for all $x\in\mathcal{X}$ and $y\in\mathcal{Y}$ :

Figures (1)

  • Figure 1: Convergence of repeated retraining approaches: (a) and (d) consider a non-robust variant of Algorithm \ref{['alg:rob_mean']} that utilizes naive gradient averaging; (b) and (c) consider Algorithm \ref{['alg:rob_mean']} that utilizes robust gradient estimation. The $y$ axis is the normalised distance between $d_{t+1}$ and $d_t$ with $c_t=1/\|d_t\|_2$, similar to the experiments in mandal2023performative, while the $x$ axis is the number of repeated retraining iterations. For Fig. \ref{['plot:fixed_epsilon_naive']} and Fig. \ref{['plot:fixed_epsilon_robust']}, $\epsilon=0.01$ and we vary $Z$. For Fig. \ref{['plot:fixed_mu_naive']} and Fig. \ref{['plot:fixed_mu_robust']}, $Z=15$ and we vary $\epsilon$. Each line presents the average over $10$ experiments initialised with different seeds.

Theorems & Definitions (30)

  • Theorem 1
  • Theorem 2
  • Remark 1
  • Theorem 3
  • Lemma 1
  • Theorem 4
  • Theorem 5
  • Theorem 6
  • proof
  • proof
  • ...and 20 more