Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Performative Policy Gradient: Optimality in Performative Reinforcement Learning

Debabrota Basu, Udvas Das, Brahim Driss, Uddalak Mukherjee

TL;DR

This work tackles performative reinforcement learning where deployed policies shift the environment, and introduces PePG, a policy-gradient method that achieves performative optimality under softmax policies with or without entropy regularisation. By proving a performative policy gradient theorem and a performative performance-difference lemma, the authors derive a gradient-based algorithm that converges to performatively optimal policies in softmax PeMDPs, with iteration complexity scaling as $ ilde{O}ig(| ext{S}|| ext{A}|^2/(oldsymbol{ ext{epsilon}}^2(1-oldsymbol{eta})^3)ig)$. The paper also contrasts optimality-seeking and stability-seeking approaches, providing both theoretical bounds and empirical validation in gridworld environments where PePG outperforms existing methods. This advances the theoretical and practical understanding of optimization under performative feedback, enabling more robust decision-making in environments that react to deployed policies. Overall, PePG offers a principled path to achieve performative optimality and demonstrates practical advantages over stability-focused alternatives in dynamic, feedback-driven RL settings.

Abstract

Post-deployment machine learning algorithms often influence the environments they act in, and thus shift the underlying dynamics that the standard reinforcement learning (RL) methods ignore. While designing optimal algorithms in this performative setting has recently been studied in supervised learning, the RL counterpart remains under-explored. In this paper, we prove the performative counterparts of the performance difference lemma and the policy gradient theorem in RL, and further introduce the Performative Policy Gradient algorithm (PePG). PePG is the first policy gradient algorithm designed to account for performativity in RL. Under softmax parametrisation, and also with and without entropy regularisation, we prove that PePG converges to performatively optimal policies, i.e. policies that remain optimal under the distribution shifts induced by themselves. Thus, PePG significantly extends the prior works in Performative RL that achieves performative stability but not optimality. Furthermore, our empirical analysis on standard performative RL environments validate that PePG outperforms standard policy gradient algorithms and the existing performative RL algorithms aiming for stability.

Performative Policy Gradient: Optimality in Performative Reinforcement Learning

TL;DR

This work tackles performative reinforcement learning where deployed policies shift the environment, and introduces PePG, a policy-gradient method that achieves performative optimality under softmax policies with or without entropy regularisation. By proving a performative policy gradient theorem and a performative performance-difference lemma, the authors derive a gradient-based algorithm that converges to performatively optimal policies in softmax PeMDPs, with iteration complexity scaling as . The paper also contrasts optimality-seeking and stability-seeking approaches, providing both theoretical bounds and empirical validation in gridworld environments where PePG outperforms existing methods. This advances the theoretical and practical understanding of optimization under performative feedback, enabling more robust decision-making in environments that react to deployed policies. Overall, PePG offers a principled path to achieve performative optimality and demonstrates practical advantages over stability-focused alternatives in dynamic, feedback-driven RL settings.

Abstract

Post-deployment machine learning algorithms often influence the environments they act in, and thus shift the underlying dynamics that the standard reinforcement learning (RL) methods ignore. While designing optimal algorithms in this performative setting has recently been studied in supervised learning, the RL counterpart remains under-explored. In this paper, we prove the performative counterparts of the performance difference lemma and the policy gradient theorem in RL, and further introduce the Performative Policy Gradient algorithm (PePG). PePG is the first policy gradient algorithm designed to account for performativity in RL. Under softmax parametrisation, and also with and without entropy regularisation, we prove that PePG converges to performatively optimal policies, i.e. policies that remain optimal under the distribution shifts induced by themselves. Thus, PePG significantly extends the prior works in Performative RL that achieves performative stability but not optimality. Furthermore, our empirical analysis on standard performative RL environments validate that PePG outperforms standard policy gradient algorithms and the existing performative RL algorithms aiming for stability.
Paper Structure (25 sections, 16 theorems, 149 equations, 6 figures, 1 table, 2 algorithms)

This paper contains 25 sections, 16 theorems, 149 equations, 6 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries: From RL to Performative RL
  3. RL: Infinite-horizon Discounted MDPs
  4. Performative RL: Infinite-horizon Discounted PeMDPs
  5. Policy Gradient Algorithms in Performative RL
  6. Impact of Policy Updates on PeMDPs
  7. Algorithm Design: Performative Policy Gradient ($\mathsf{PePG}$)
  8. Convergence Analysis of $\mathsf{PePG}$: Softmax Policies and Softmax PeMDPs
  9. Experimental Analysis
  10. Discussions, Limitations, and Future Works
  11. Appendix
  12. Notations
  13. Details of the Toy Example: Loan Approvement Problem
  14. Extended Related Works
  15. Impact of Policy Updates on PeMDPs (Section \ref{['sec:analysis']})
  16. ...and 10 more sections

Key Result

Theorem 1

Fix a differentiable paramterisation $\theta \mapsto \pi_\theta(a \mid s)$ and an initial distribution ${\boldsymbol\rho}$. Let us define the Q-value function $Q^{{\boldsymbol\pi}_{\boldsymbol\theta}}(s, a) \triangleq \mathbb{E}_{s_{t+1} \sim \mathbf{P}_{{\boldsymbol\pi}}(\cdot \mid s_t, {\boldsymbo

Figures (6)

  • Figure 1: Average reward (over 10 runs) obtained by ERM and Performative Optimal policies across performative strength $\beta$.
  • Figure 2: Comparison of evolution in expected average return (both regularised and unregularised) and stability of $\mathsf{PePG}$ with SOTA stability-achieving methods. Each algorithm is run for 20 random seeds and 1000 iterations.
  • Figure 3: Ablation study for $\mathsf{PePG}$ for different values of regularised $\lambda$ with 20 random seeds, each for 100 iterations
  • Figure 4: Ablation study for $\mathsf{PePG}$ for different values of $\eta$ with 20 random seeds across 100 iterations
  • Figure : Vanilla Policy Gradient
  • ...and 1 more figures

Theorems & Definitions (43)

  • Example 1: Performative RL in loan approval
  • Theorem 1: Policy Gradient Theorem Sutton1999
  • Definition 1: Performative Value Function
  • Definition 2: Performative Q-value
  • Definition 3: Performative Optimality
  • Definition 4: Performative Stability
  • Definition 5: Entropy Regularised (or Soft) Performative Value Function
  • Lemma 1: Performative Performance Difference Lemma
  • Lemma 2: Bounding Performative Performance Difference for Gradually Shifting Environments
  • Theorem 2: Performative Policy Gradient Theorem
  • ...and 33 more