Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Performative Learning Theory

Julian Rodemann, Unai Fischer-Abaigar, James Bailie, Krikamol Muandet

TL;DR

This work formalizes generalization under performativity, where predictive models influence the data they seek to predict, by embedding performative effects into statistical learning theory via Wasserstein-based min-max and min-min risk functionals. It treats performativity on the training sample, the population, or both, deriving nonasymptotic generalization bounds under minimal regularity and Lipschitz assumptions, and revealing a fundamental trade-off: greater performative influence on data can reduce learnability. The authors provide a sequence of results (RQ1–RQ3) giving excess-risk and cumulative-excess-risk bounds, plus a corollary showing how retraining on distorted samples can tighten guarantees. They illustrate the bounds in a case study on unemployment prediction in Germany, showing how performative effects on job trainings complicate generalization and how bounds decompose into sampling, complexity, and performative terms. Overall, the paper offers principled guidance for understanding and mitigating the impact of performativity on learnability in high-stakes decision contexts.

Abstract

Performative predictions influence the very outcomes they aim to forecast. We study performative predictions that affect a sample (e.g., only existing users of an app) and/or the whole population (e.g., all potential app users). This raises the question of how well models generalize under performativity. For example, how well can we draw insights about new app users based on existing users when both of them react to the app's predictions? We address this question by embedding performative predictions into statistical learning theory. We prove generalization bounds under performative effects on the sample, on the population, and on both. A key intuition behind our proofs is that in the worst case, the population negates predictions, while the sample deceptively fulfills them. We cast such self-negating and self-fulfilling predictions as min-max and min-min risk functionals in Wasserstein space, respectively. Our analysis reveals a fundamental trade-off between performatively changing the world and learning from it: the more a model affects data, the less it can learn from it. Moreover, our analysis results in a surprising insight on how to improve generalization guarantees by retraining on performatively distorted samples. We illustrate our bounds in a case study on prediction-informed assignments of unemployed German residents to job trainings, drawing upon administrative labor market records from 1975 to 2017 in Germany.

Performative Learning Theory

TL;DR

This work formalizes generalization under performativity, where predictive models influence the data they seek to predict, by embedding performative effects into statistical learning theory via Wasserstein-based min-max and min-min risk functionals. It treats performativity on the training sample, the population, or both, deriving nonasymptotic generalization bounds under minimal regularity and Lipschitz assumptions, and revealing a fundamental trade-off: greater performative influence on data can reduce learnability. The authors provide a sequence of results (RQ1–RQ3) giving excess-risk and cumulative-excess-risk bounds, plus a corollary showing how retraining on distorted samples can tighten guarantees. They illustrate the bounds in a case study on unemployment prediction in Germany, showing how performative effects on job trainings complicate generalization and how bounds decompose into sampling, complexity, and performative terms. Overall, the paper offers principled guidance for understanding and mitigating the impact of performativity on learnability in high-stakes decision contexts.

Abstract

Performative predictions influence the very outcomes they aim to forecast. We study performative predictions that affect a sample (e.g., only existing users of an app) and/or the whole population (e.g., all potential app users). This raises the question of how well models generalize under performativity. For example, how well can we draw insights about new app users based on existing users when both of them react to the app's predictions? We address this question by embedding performative predictions into statistical learning theory. We prove generalization bounds under performative effects on the sample, on the population, and on both. A key intuition behind our proofs is that in the worst case, the population negates predictions, while the sample deceptively fulfills them. We cast such self-negating and self-fulfilling predictions as min-max and min-min risk functionals in Wasserstein space, respectively. Our analysis reveals a fundamental trade-off between performatively changing the world and learning from it: the more a model affects data, the less it can learn from it. Moreover, our analysis results in a surprising insight on how to improve generalization guarantees by retraining on performatively distorted samples. We illustrate our bounds in a case study on prediction-informed assignments of unemployed German residents to job trainings, drawing upon administrative labor market records from 1975 to 2017 in Germany.
Paper Structure (33 sections, 10 theorems, 132 equations, 2 figures, 1 table)

This paper contains 33 sections, 10 theorems, 132 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Running Examples
  3. Generalization Under Performativity
  4. On the Meaning of Performative Generalization
  5. Performative Generalization Bounds
  6. RQ1: Generalization Under Sample Performativity
  7. RQ2: Generalization Under Full Performativity
  8. Cumulative Performative Excess Risk Bound, RQ3
  9. Illustration of Bounds on Job Seeker Data
  10. Historical Data (Sample Performativity, RQ1):
  11. Semi-Simulation (Full Performativity, RQ2):
  12. Related Work, Limitations, Discussion and Outlook
  13. Related work
  14. Limitations and General Discussion
  15. Outlook
  16. ...and 18 more sections

Key Result

Lemma 3.4

Let $\mathscr D_{\mathcal{Z}} := \sup_{z, z'} \|z - z'\|_2 < \infty$. Then, for any $\beta_0 \in(0, \infty)$, we have $W_{p}\left(\widehat{d}_0, d_0\right) \leq \beta_0,$ with probability at least $C_{a} \exp \left(-C_{b} n \beta_0^{\nu / p}\right)$, where $C_{a}$ and $C_{b}$ are constants which dep

Figures (2)

  • Figure 1: 1.1Example (A): Route predictions are known to have performative effects: Drivers avoid routes with predicted congestion, thereby rendering these predictions less accurate. Can routing apps still generalize from San Francisco (sample) to the whole Bay Area (population)? 1.2Example (B): A job center in Bavaria assigns job training programs to those among the unemployed that have high risks of long-term unemployment according to ML predictions. As a result of the job training, their probability of finding a job increases, a textbook example of a performative effect, see Section \ref{['sec:job-trainings']}. Can the job center's ML model trained on performatively shifted data from Bavaria generalize to the whole German population, which will in turn react to the predictions? 1.3 Theorem \ref{['thm:gen-gap-stronger-ass']} applied to Jobseekers: Growth of generalization gap bound (blue, logistic loss units) shows trade-off between performatively changing a population (by assigning more people to job trainings) and reliably learning its properties. Details on the bound's components (red, green, yellow) can be found in Section \ref{['sec:job-trainings']}.
  • Figure 2: Generalization gap bound from Theorem \ref{['thm:gen-gap-stronger-ass-2']} and its decomposition as a function of $\xi$ (fraction of jobseekers receiving training). The total bound (blue) is decomposed into complexity term $\mathsf{Comp}_B(\xi)$, sampling term $\mathsf{Samp}(\xi)$, and performative term $\mathsf{Perf}(\xi)$ (see details in Appendix \ref{['app:details-case-study']}. All values are in logistic loss units. Parameters: $n$ training samples, $B=10^{-3}$, $\delta=0.05$, $p=2$.

Theorems & Definitions (23)

  • Definition 2.1: Risk (Minimizer) and Loss
  • Definition 2.2: Excess Risk
  • Definition 2.3: Repeated (Empirical) Risk Minimization
  • Lemma 3.4: Convergence in Wasserstein Space; fournier2015rate
  • Lemma 3.5: In-Sample Performative Shift Bound
  • Definition 3.6: Covering Number Entropy Integrals
  • Theorem 3.7: Excess Risk Bound, \ref{['rq:1']}
  • Corollary 3.8
  • Lemma 3.9: Performative Population Shift Bound
  • Theorem 3.10: Performative Excess Risk Bound, \ref{['rq:2']}
  • ...and 13 more