Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conditional Forecasts and Proper Scoring Rules for Reliable and Accurate Performative Predictions

Philip Boeken, Onno Zoeter, Joris M. Mooij

TL;DR

The paper studies performative forecasting, where forecasts influence the outcomes they predict, and shows that conditioning on covariates that separate the forecast from the target restores a well-posed problem via forecast-invariance $P_M(Y\mid A, do(F)) = P_M(Y\mid A)$. It proves that classical scoring rules are generally not performatively proper and offers two remedies: (i) a decision-theoretic approach that achieves incentive compatibility under forecast separability, and (ii) a divergence-based scoring method using unbiased estimators of the performative divergence $D_{ ext{pc}}(F,M)$, which yields correct forecasts. The authors further develop parametric estimation in performative settings, demonstrating performative stability and optimality when using divergence-based scoring, and provide finite-sample unbiased estimators for key divergences (Brier, Energy) in binary/continuous outcomes. Together, these results delineate the limits of classical forecast evaluation in performative contexts and propose practical tools for reliable, stable, and accurate performative forecasts and parameter estimation. The work connects to prior literature on self-fulfilling prophecies and Perdomo et al. (2020) by offering principled scoring and estimation strategies under causal guidance.

Abstract

Performative predictions are forecasts which influence the outcomes they aim to predict, undermining the existence of correct forecasts and standard methods of elicitation and estimation. We show that conditioning forecasts on covariates that separate them from the outcome renders the target distribution forecast-invariant, guaranteeing well-posedness of the forecasting problem. However, even under this condition, classical proper scoring rules fail to elicit correct forecasts. We prove a general impossibility result and identify two solutions: (i) in decision-theoretic settings, elicitation of correct and incentive-compatible forecasts is possible if forecasts are separating; (ii) scoring with unbiased estimates of the divergence between the forecast and the induced distribution of the target variable yields correct forecasts. Applying these insights to parameter estimation, conditional forecasts and proper scoring rules enable performatively stable estimation of performatively correct parameters, resolving the issues raised by Perdomo et al. (2020). Our results expose fundamental limits of classical forecast evaluation and offer new tools for reliable and accurate forecasting in performative settings.

Conditional Forecasts and Proper Scoring Rules for Reliable and Accurate Performative Predictions

TL;DR

The paper studies performative forecasting, where forecasts influence the outcomes they predict, and shows that conditioning on covariates that separate the forecast from the target restores a well-posed problem via forecast-invariance . It proves that classical scoring rules are generally not performatively proper and offers two remedies: (i) a decision-theoretic approach that achieves incentive compatibility under forecast separability, and (ii) a divergence-based scoring method using unbiased estimators of the performative divergence , which yields correct forecasts. The authors further develop parametric estimation in performative settings, demonstrating performative stability and optimality when using divergence-based scoring, and provide finite-sample unbiased estimators for key divergences (Brier, Energy) in binary/continuous outcomes. Together, these results delineate the limits of classical forecast evaluation in performative contexts and propose practical tools for reliable, stable, and accurate performative forecasts and parameter estimation. The work connects to prior literature on self-fulfilling prophecies and Perdomo et al. (2020) by offering principled scoring and estimation strategies under causal guidance.

Abstract

Performative predictions are forecasts which influence the outcomes they aim to predict, undermining the existence of correct forecasts and standard methods of elicitation and estimation. We show that conditioning forecasts on covariates that separate them from the outcome renders the target distribution forecast-invariant, guaranteeing well-posedness of the forecasting problem. However, even under this condition, classical proper scoring rules fail to elicit correct forecasts. We prove a general impossibility result and identify two solutions: (i) in decision-theoretic settings, elicitation of correct and incentive-compatible forecasts is possible if forecasts are separating; (ii) scoring with unbiased estimates of the divergence between the forecast and the induced distribution of the target variable yields correct forecasts. Applying these insights to parameter estimation, conditional forecasts and proper scoring rules enable performatively stable estimation of performatively correct parameters, resolving the issues raised by Perdomo et al. (2020). Our results expose fundamental limits of classical forecast evaluation and offer new tools for reliable and accurate forecasting in performative settings.
Paper Structure (36 sections, 27 theorems, 47 equations, 10 figures)

This paper contains 36 sections, 27 theorems, 47 equations, 10 figures.

Table of Contents

  1. Introduction
  2. Related work and detailed contributions
  3. Existence of correct forecasts through conditioning
  4. Eliciting correct performative forecasts with proper scoring rules
  5. Defining properness of scoring rules for performative forecasts
  6. Impossibility of eliciting correct performative forecasts with classical scoring rules
  7. Properness and incentive compatibility in a decision-theoretic setting
  8. Eliciting correct forecasts with estimates of the performative divergence
  9. Parameter estimation based on scoring rules
  10. Discussion
  11. Introduction to causal models
  12. Graphs
  13. Bayesian network
  14. Structural Causal Models
  15. Markov property and faithfulness
  16. ...and 21 more sections

Key Result

theorem 1

Let $\mathcal{M}_G'\subseteq \mathcal{M}_{\mathrm{pc}}$ be the set of models compatibleMeaning that the Markov property holds: $A\mathop{\mathrm{\perp}}\limits^d_G B\mathop{\mathrm{|}}\limits C \implies A\mathop{\mathrm{\perp\mkern-11mu\perp}}\limits_{P_M} B \mathop{\mathrm{|}}\limits C$ for all set

Figures (10)

  • Figure 1:
  • Figure 2:
  • Figure 3: Example of DAG $G$ and a latent projection.
  • Figure 4: Merging of variables $A_1, A_2, A_3$ of the graph in Figure \ref{['fig:latent_projection']} to a single vertex $A$.
  • Figure 5: Causal graph of performative forecast depending on covariates $X$.
  • ...and 5 more figures

Theorems & Definitions (69)

  • Definition 1
  • Example 1
  • Example 2
  • Definition 2
  • Definition 3
  • Example 3
  • theorem 1
  • Definition 4
  • Definition 5
  • Example 4
  • ...and 59 more