Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PAC Learnability in the Presence of Performativity

Ivan Kirev, Lyuben Baltadzhiev, Nikola Konstantinov

TL;DR

The paper addresses PAC learnability under model-induced distribution shifts caused by performativity, formalizing a performative PAC framework relative to a distribution map $\tilde{\mathcal{D}}(\cdot, \mathcal{D})$ and introducing a linear posterior drift family. It derives the performative empirical risk $PR_n(h)$ as an unbiased estimator of the true performative risk $PR(h)$ and shows that minimizing $PR_n(h)$ yields a PAC learner whenever the standard binary-classification class is PAC-learnable, even under linear drift. Generalization bounds are obtained via Rademacher complexity, with extensions to imperfect information about the distribution map and to broader performative maps using density-ratio reweighting. Empirically, PERM improves performance over ERM across synthetic data and real-world credit/income datasets, and analysis of RERM illustrates both its potential and its limitations under certain shift dynamics. The work offers a principled statistical approach to learning under self-induced distribution shifts and points to future work on richer performative maps and real-world experimental validation.

Abstract

Following the wide-spread adoption of machine learning models in real-world applications, the phenomenon of performativity, i.e. model-dependent shifts in the test distribution, becomes increasingly prevalent. Unfortunately, since models are usually trained solely based on samples from the original (unshifted) distribution, this performative shift may lead to decreased test-time performance. In this paper, we study the question of whether and when performative binary classification problems are learnable, via the lens of the classic PAC (Probably Approximately Correct) learning framework. We motivate several performative scenarios, accounting in particular for linear shifts in the label distribution, as well as for more general changes in both the labels and the features. We construct a performative empirical risk function, which depends only on data from the original distribution and on the type performative effect, and is yet an unbiased estimate of the true risk of a classifier on the shifted distribution. Minimizing this notion of performative risk allows us to show that any PAC-learnable hypothesis space in the standard binary classification setting remains PAC-learnable for the considered performative scenarios. We also conduct an extensive experimental evaluation of our performative risk minimization method and showcase benefits on synthetic and real data.

PAC Learnability in the Presence of Performativity

TL;DR

The paper addresses PAC learnability under model-induced distribution shifts caused by performativity, formalizing a performative PAC framework relative to a distribution map and introducing a linear posterior drift family. It derives the performative empirical risk as an unbiased estimator of the true performative risk and shows that minimizing yields a PAC learner whenever the standard binary-classification class is PAC-learnable, even under linear drift. Generalization bounds are obtained via Rademacher complexity, with extensions to imperfect information about the distribution map and to broader performative maps using density-ratio reweighting. Empirically, PERM improves performance over ERM across synthetic data and real-world credit/income datasets, and analysis of RERM illustrates both its potential and its limitations under certain shift dynamics. The work offers a principled statistical approach to learning under self-induced distribution shifts and points to future work on richer performative maps and real-world experimental validation.

Abstract

Following the wide-spread adoption of machine learning models in real-world applications, the phenomenon of performativity, i.e. model-dependent shifts in the test distribution, becomes increasingly prevalent. Unfortunately, since models are usually trained solely based on samples from the original (unshifted) distribution, this performative shift may lead to decreased test-time performance. In this paper, we study the question of whether and when performative binary classification problems are learnable, via the lens of the classic PAC (Probably Approximately Correct) learning framework. We motivate several performative scenarios, accounting in particular for linear shifts in the label distribution, as well as for more general changes in both the labels and the features. We construct a performative empirical risk function, which depends only on data from the original distribution and on the type performative effect, and is yet an unbiased estimate of the true risk of a classifier on the shifted distribution. Minimizing this notion of performative risk allows us to show that any PAC-learnable hypothesis space in the standard binary classification setting remains PAC-learnable for the considered performative scenarios. We also conduct an extensive experimental evaluation of our performative risk minimization method and showcase benefits on synthetic and real data.

Paper Structure

This paper contains 30 sections, 14 theorems, 52 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Framework
  4. Basic Setup
  5. Definition rationale
  6. Linear Posterior Performative Drift
  7. PAC Learnability under the Linear Performative Posterior Drift
  8. Performative Empirical Risk
  9. Generalization Bounds
  10. Imperfect Information about the Distribution Map
  11. Learnability under Other Performative Maps
  12. Experiments
  13. Minimizing PER
  14. Experiments
  15. Repeated Empirical Risk Minimization
  16. ...and 15 more sections

Key Result

Lemma 4.1

Let $S = \{ (X_i, Y_i) \}_{i=1}^n$ be a set of $n$ i.i.d. samples from $\mathcal{D}$, and let $h \in \mathcal{H}$ be a hypothesis. Suppose $\tilde{\mathcal{D}}(h)$ is the distribution obtained from $\mathcal{D}$ after a performative linear shift as defined in eq: distribution_map. Then the performat Moreover, an unbiased estimator of $\text{\normalfont PR}(h)$ is the performative empirical risk w

Figures (3)

  • Figure 1: Accuracy on the performative test set as a function of effect strength $a$.
  • Figure 2: Behavior of the RERM-inspired method on the credit score dataset.
  • Figure 3: Experiments on the models 1 - 5 for the three datasets: Synthetic, Credit Score, and Folktables.

Theorems & Definitions (27)

  • Definition 3.1
  • Remark 3.1
  • Lemma 4.1
  • Remark 4.1
  • Remark 4.2
  • Definition 4.1: Rademacher Averages
  • Theorem 4.1
  • Corollary 4.1
  • Theorem 4.2
  • Proposition 4.1
  • ...and 17 more