Practical Performative Policy Learning with Strategic Agents

TL;DR

The paper addresses the challenge of performative policy learning where agents adapt their features in response to a released policy, causing endogenous distribution shifts. It drops strong parametric models and instead leverages bounded rationality to reduce the intervention to a low-dimensional mediator, the evaluation vector $\zeta(v, \pi_\theta)$, while modeling agent behavior with a differentiable classifier $h_\gamma$. A gradient-based strategic policy gradient algorithm combines CATE estimation with a nonparametric performative gradient via $p(u|\zeta)$, and comes with convergence guarantees grounded in RKHS realizability. Empirically, it achieves high sample efficiency and robustness in high-dimensional synthetic and semi-synthetic settings, outperforming baselines and demonstrating that discretization of manipulatable features can be incentivized without substantial loss of accuracy. This approach offers a practical path for deploying policies in strategic environments like lending or admissions, where agents’ genuine effort to modify features drives endogenous shifts.

Abstract

This paper studies the performative policy learning problem, where agents adjust their features in response to a released policy to improve their potential outcomes, inducing an endogenous distribution shift. There has been growing interest in training machine learning models in strategic environments, including strategic classification and performative prediction. However, existing approaches often rely on restrictive parametric assumptions: micro-level utility models in strategic classification and macro-level data distribution maps in performative prediction, severely limiting scalability and generalizability. We approach this problem as a complex causal inference task, relaxing parametric assumptions on both micro-level agent behavior and macro-level data distribution. Leveraging bounded rationality, we uncover a practical low-dimensional structure in distribution shifts and construct an effective mediator in the causal path from the deployed model to the shifted data. We then propose a gradient-based policy optimization algorithm with a differentiable classifier as a substitute for the high-dimensional distribution map. Our algorithm efficiently utilizes batch feedback and limited manipulation patterns. Our approach achieves high sample efficiency compared to methods reliant on bandit feedback or zero-order optimization. We also provide theoretical guarantees for algorithmic convergence. Extensive and challenging experiments on high-dimensional settings demonstrate our method's practical efficacy.

Figures (11)

• Figure 1: Default case
• Figure 2: With proposed mediator
• Figure 4: The left figure presents the curve of policy value in synthetic experiment with $c=0.1$, and right figure presents the original distribution of $u$ and that after manipulation induced by policies.
• Figure 5: The left figure is the curve of policy value with noisy utility, and the right figure is that with softmax manipulation mechanism.
• Figure 6: The left figure presents policy value curves of the proposed strategic policy gradient with MLP and Gaussian process classifier as the behavior model. The right figure illustrates the curve of the loss functions the behavior model, i.e. cross-entropy and negative ELBO, respectively.

Theorems & Definitions (5)

• Proposition 1
• Lemma 1
• Theorem 1: Convergence of strategic policy gradient
• Proposition 2
• Lemma 2