Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Falsifying Predictive Algorithm

Amanda Coston

TL;DR

This work introduces a falsification framework for predictive algorithms to test discriminant validity, i.e., whether a model predicts the intended unobservable outcome $Y^{*}$ better than impermissible proxies. It relies on calibrating predictions with Platt scaling, comparing calibrated prediction losses across permissible proxies and impermissible ones, and applying nonparametric tests (t-test or Wilcoxon; permutation or normal-approximation) to assess loss discriminance or exchangeability. The method is demonstrated in two real-world domains: law school admissions (LSAC data) and COMPAS recidivism prediction, revealing discriminant validity for gender but not for race in admissions, and highlighting limitations and the need for complementary approaches in criminal justice settings. The framework emphasizes that passing falsification checks does not guarantee fairness or readiness for deployment, but provides a principled validity check to prompt re-formulation of problems, data, and task design before proceeding to further analyses. Overall, it advocates validity-centered evaluation as a proactive safeguard against deploying models that predict unintended or impermissible quantities.

Abstract

Empirical investigations into unintended model behavior often show that the algorithm is predicting another outcome than what was intended. These exposes highlight the need to identify when algorithms predict unintended quantities - ideally before deploying them into consequential settings. We propose a falsification framework that provides a principled statistical test for discriminant validity: the requirement that an algorithm predict intended outcomes better than impermissible ones. Drawing on falsification practices from causal inference, econometrics, and psychometrics, our framework compares calibrated prediction losses across outcomes to assess whether the algorithm exhibits discriminant validity with respect to a specified impermissible proxy. In settings where the target outcome is difficult to observe, multiple permissible proxy outcomes may be available; our framework accommodates both this setting and the case with a single permissible proxy. Throughout we use nonparametric hypothesis testing methods that make minimal assumptions on the data-generating process. We illustrate the method in an admissions setting, where the framework establishes discriminant validity with respect to gender but fails to establish discriminant validity with respect to race. This demonstrates how falsification can serve as an early validity check, prior to fairness or robustness analyses. We also provide analysis in a criminal justice setting, where we highlight the limitations of our framework and emphasize the need for complementary approaches to assess other aspects of construct validity and external validity.

Falsifying Predictive Algorithm

TL;DR

This work introduces a falsification framework for predictive algorithms to test discriminant validity, i.e., whether a model predicts the intended unobservable outcome better than impermissible proxies. It relies on calibrating predictions with Platt scaling, comparing calibrated prediction losses across permissible proxies and impermissible ones, and applying nonparametric tests (t-test or Wilcoxon; permutation or normal-approximation) to assess loss discriminance or exchangeability. The method is demonstrated in two real-world domains: law school admissions (LSAC data) and COMPAS recidivism prediction, revealing discriminant validity for gender but not for race in admissions, and highlighting limitations and the need for complementary approaches in criminal justice settings. The framework emphasizes that passing falsification checks does not guarantee fairness or readiness for deployment, but provides a principled validity check to prompt re-formulation of problems, data, and task design before proceeding to further analyses. Overall, it advocates validity-centered evaluation as a proactive safeguard against deploying models that predict unintended or impermissible quantities.

Abstract

Empirical investigations into unintended model behavior often show that the algorithm is predicting another outcome than what was intended. These exposes highlight the need to identify when algorithms predict unintended quantities - ideally before deploying them into consequential settings. We propose a falsification framework that provides a principled statistical test for discriminant validity: the requirement that an algorithm predict intended outcomes better than impermissible ones. Drawing on falsification practices from causal inference, econometrics, and psychometrics, our framework compares calibrated prediction losses across outcomes to assess whether the algorithm exhibits discriminant validity with respect to a specified impermissible proxy. In settings where the target outcome is difficult to observe, multiple permissible proxy outcomes may be available; our framework accommodates both this setting and the case with a single permissible proxy. Throughout we use nonparametric hypothesis testing methods that make minimal assumptions on the data-generating process. We illustrate the method in an admissions setting, where the framework establishes discriminant validity with respect to gender but fails to establish discriminant validity with respect to race. This demonstrates how falsification can serve as an early validity check, prior to fairness or robustness analyses. We also provide analysis in a criminal justice setting, where we highlight the limitations of our framework and emphasize the need for complementary approaches to assess other aspects of construct validity and external validity.
Paper Structure (33 sections, 2 figures, 5 tables, 2 algorithms)

This paper contains 33 sections, 2 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Validity in Algorithmic Systems.
  4. Proxy Outcomes and Label Bias.
  5. Algorithmic Fairness Evaluation.
  6. Falsification in Other Domains
  7. Falsification in causal inference
  8. Falsification in Measurement
  9. Problem Formulation
  10. Falsifying Predictive Algorithms
  11. Method
  12. Falsification with a Single Permissible Proxy
  13. Falsification with Multiple Permissible Proxies
  14. Choice of Loss Function.
  15. Multiple Hypothesis Testing
  16. ...and 18 more sections

Figures (2)

  • Figure 1: Rank distribution of impermissible proxies (gender and race) for Alg. 2 on the LSAC dataset. For each observation, we rank the calibrated log-loss for the impermissible proxy against the three permissible proxies (cumulative GPA, first-year GPA, bar passage). Rank 1 = best predicted; Rank 4 = worst predicted. The dashed line shows the expected 25% under the null hypothesis. Left: Gender is worst-predicted (rank 4) for 45.7% of observations, and the model passes our test ($p \approx 0$) indicating discriminant validity with respect to gender. Right: Race is best-predicted (rank 1) for 91% of observations, indicating the model predicts race better than permissible outcomes. The model fails to pass our test of discriminant validity with respect to race ($p \approx 1$).
  • Figure 2: Paired difference distributions for Alg. 1 on COMPAS with age (left) and race (right) as impermissible proxies. Histograms show the distribution of (re-arrest loss $-$ impermissible proxy loss) across observations. Dashed black lines mark zero. Left (Age): Fail to establish discriminant validity wrt age ($p \approx 1$). Right (Race): Borderline result for discriminant validity wrt ($p = 0.025$).

Theorems & Definitions (3)

  • Definition 4.1
  • Definition 4.2
  • Definition 4.3