Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal Regularization for Performative Learning

Edwige Cyffers, Alireza Mirrokni, Marco Mondelli

TL;DR

This work studies how ridge regularization interacts with performative effects in regression under repeated retraining. By analyzing both population and over-parameterized regimes, the authors derive how the optimal regularization scales with performative strength and show that regularization can mitigate risk in the presence of performativity, while in high-dimensional settings it can even improve risk when performativity reinforces predictive trends. They provide explicit formulas for the optimal ridge in the population setting and deterministic equivalents in the high-dimensional regime, plus expansions that reveal sign patterns depending on noise and data structure. Empirical results on synthetic data and real-world datasets (Housing, LSAC) validate the theoretical insights and illustrate practical guidelines for choosing regularization in performative contexts.

Abstract

In performative learning, the data distribution reacts to the deployed model - for example, because strategic users adapt their features to game it - which creates a more complex dynamic than in classical supervised learning. One should thus not only optimize the model for the current data but also take into account that the model might steer the distribution in a new direction, without knowing the exact nature of the potential shift. We explore how regularization can help cope with performative effects by studying its impact in high-dimensional ridge regression. We show that, while performative effects worsen the test risk in the population setting, they can be beneficial in the over-parameterized regime where the number of features exceeds the number of samples. We show that the optimal regularization scales with the overall strength of the performative effect, making it possible to set the regularization in anticipation of this effect. We illustrate this finding through empirical evaluations of the optimal regularization parameter on both synthetic and real-world datasets.

Optimal Regularization for Performative Learning

TL;DR

This work studies how ridge regularization interacts with performative effects in regression under repeated retraining. By analyzing both population and over-parameterized regimes, the authors derive how the optimal regularization scales with performative strength and show that regularization can mitigate risk in the presence of performativity, while in high-dimensional settings it can even improve risk when performativity reinforces predictive trends. They provide explicit formulas for the optimal ridge in the population setting and deterministic equivalents in the high-dimensional regime, plus expansions that reveal sign patterns depending on noise and data structure. Empirical results on synthetic data and real-world datasets (Housing, LSAC) validate the theoretical insights and illustrate practical guidelines for choosing regularization in performative contexts.

Abstract

In performative learning, the data distribution reacts to the deployed model - for example, because strategic users adapt their features to game it - which creates a more complex dynamic than in classical supervised learning. One should thus not only optimize the model for the current data but also take into account that the model might steer the distribution in a new direction, without knowing the exact nature of the potential shift. We explore how regularization can help cope with performative effects by studying its impact in high-dimensional ridge regression. We show that, while performative effects worsen the test risk in the population setting, they can be beneficial in the over-parameterized regime where the number of features exceeds the number of samples. We show that the optimal regularization scales with the overall strength of the performative effect, making it possible to set the regularization in anticipation of this effect. We illustrate this finding through empirical evaluations of the optimal regularization parameter on both synthetic and real-world datasets.
Paper Structure (23 sections, 15 theorems, 107 equations, 4 figures, 1 table)

This paper contains 23 sections, 15 theorems, 107 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Performative learning.
  4. High-dimensional regression and role of ridge regularization.
  5. Preliminaries and problem setup
  6. Analysis in the population setting
  7. Excess risk at the performative fixed point.
  8. Optimal regularization and optimally regularized risk.
  9. Analysis in the over-parameterized setting
  10. Deterministic equivalent of the performative fixed point.
  11. Optimal regularization and optimally regularized risk.
  12. Numerical experiments
  13. Conclusions
  14. Additional proofs for \ref{['sec:pop']}
  15. Proof of Theorem \ref{['thm:over']}
  16. ...and 8 more sections

Key Result

Theorem 1

Let $F = D - \lambda \Sigma^{-1}$. Then, we have where $\|\cdot\|_{\mathrm{op}}$ denotes the operator norm, $\bar{b} := \frac{1}{d}\operatorname{Tr}[\operatorname{diag}(b)] = \frac{1}{d}\sum_{i=1}^d b_i$, $b^2:=[b_1^2, \ldots, b_d^2]\in\mathbb R^d$ and $S_1 = (\Sigma_1 - \Sigma_{12}\Sigma_2^{-1} \Sigma_{21})^{-1}$ denotes the Schur complement of $

Figures (4)

  • Figure 1: Excess risk at the performative fixed point $\theta^{\infty}$ in (\ref{['eq:fppop']}), as a function of ridge regularization $\lambda$, for $d = 100$, $\Sigma=I_p$, entries of $b$ uniform in $\left[\min\{0, 2 \bar{b}\}, \max\{0, 2 \bar{b}\} \right]$, $c=0$ and $\sigma = 0.1$. Empirical values (blue dots) are computed from 20 i.i.d. trials on $a$ and 5 i.i.d. trials on $b$, with error band at 1 standard deviation. Theoretical predictions (blue dashed curves) are from (\ref{['eq:fppopavg']}) and match perfectly empirical values. First-order approximations (orange dashed curves) are given by $\widetilde{\mathcal{R}}_{\rm pop}(D, \lambda, \Sigma)$ in (\ref{['eq:popapex']}) and still provide a good match when $\lambda$ is near-optimal. The green vertical line is the optimal regularization obtained by numerically optimizing the excess risk of $\theta^\infty$ ($\lambda^*_{\rm emp}$), the red one is the first-order approximation ($\lambda_{\rm pop}^*$ from (\ref{['eq:optpopap1']})) and the violet one the second-order approximation ($\lambda^*_{{\rm pop}, 2}$ minimizing (\ref{['eq:pophigh']})).
  • Figure 2: Optimal regularization and risk for the performative fixed point $\theta^{\infty}$ in (\ref{['eq:fppop']}), with $d = 100$, $\Sigma_1=\Sigma_2=I_d$, $\Sigma_{12}=\rho I_d$. Values are computed from 20 i.i.d. trials on $a$ and 5 i.i.d. trials on $b$, with error band at 1 standard deviation. (a) Optimal regularization as a function of $\bar{b}$ for $\rho\in \{0, 0.4, 0.8\}$. The entries of $b$ are uniform in $\left[\min\{0, 2 \bar{b}\}, \max\{0, 2 \bar{b}\} \right]$ and $c = 0$. (b) Optimal risk as a function of $\bar{b}$. Different curves correspond to different variances $\sigma_b^2$ of the entries of $b$, which are uniform in $\left[\bar{b} - \sigma_b \sqrt{3}, \bar{b} + \sigma_b \sqrt{3} \right]$ for $\sigma_b\in\{0, 0.2, 0.4\}$. We pick $c=0$ and $\rho=0$. We note that, when $\rho=0$, ${\mathcal{R}}_{\rm pop}^*(D, \Sigma)$ equals the empirical variance of the entries of $b$ and, as such, it does not depend on $\bar{b}$. (c) Optimal regularization as a function of $\bar{b}$ for $\bar{c}\in \{-0.3, 0, 0.3\}$. The entries of $b$ are uniform in $\left[\min\{0, 2 \bar{b}\}, \max\{0, 2 \bar{b}\} \right]$, the entries of $c$ are uniform in $\left[\min\{0, 2 \bar{c}\}, \max\{0, 2 \bar{c}\} \right]$, and $\rho = 0.5$.
  • Figure 3: Excess risk as a function of ridge regularization $\lambda$ with Gaussian data, for $n = 4000$, $\Sigma_1=\Sigma_2=I_d$, $\Sigma_{12}=\rho I_d$, entries of $b$ equal to $\bar{b}$, and entries of $c$ equal to $\bar{c}$. Values are computed from 20 i.i.d. trials, with error band at 1 standard deviation. We perform $5$ steps of RRM to approximate the fixed point, as in the simulation setup of Section \ref{['sec:num']}. (a) In the low-noise regime ($\sigma=0.2$), taking $\bar{b}=0.2$ instead of $\bar{b}=0$increases the optimal regularization and reduces the optimal risk. We set $\bar{c}=0$ to emphasize the dependence on $\bar{b}$. (b) In the large-noise regime ($\sigma=0.7$), taking $\bar{b}=0.2$ instead of $\bar{b}=0$reduces both optimal regularization and optimal risk. As in (a), we set $\bar{c}=0$. (c) Taking $\bar{c}=0.2$ instead of $\bar{c}=0$reduces the optimal risk, although the the impact of $\bar{c}$ is less pronounced. We set $\bar{b}=0$ to emphasize the dependence on $\bar{c}$.
  • Figure 4: Excess risk as a function of ridge regularization $\lambda$ in real-world datasets (Housing, LSAC). Different curves (in different colors) correspond to different values of $\bar{b} \in \{0, 0.05, 0.1, 0.15, 0.2\}$, and we connect with a red dashed line the optima of the risk for various choices of $\bar{b}$. The plots in (a)-(b) use $n=4000$ data points at each training step, which corresponds to the population setting ($n \gg d$); the plot in (c) uses $n=100$, a value closer to the number of features $d=22$.

Theorems & Definitions (27)

  • Theorem 1: Excess risk -- population
  • Corollary 2: Optimal regularization -- population
  • Theorem 3: Excess risk -- over-parameterized
  • Theorem 4: Optimal regularization -- over-parameterized
  • Lemma 5
  • proof
  • proof : Proof of Theorem \ref{['thm:pop']} and of the higher-order approximation in (\ref{['eq:pophigh']})
  • Lemma 6
  • proof
  • Lemma 7
  • ...and 17 more