Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do We Really Even Need Data?

Kentaro Hoffman, Stephen Salerno, Awan Afiaz, Jeffrey T. Leek, Tyler H. McCormick

TL;DR

The statistical challenges inherent to this so-called ``inference with predicted data'' problem are characterized and three potential sources of error are elucidated, including the relationship between predicted outcomes and their true, unobserved counterparts.

Abstract

As artificial intelligence and machine learning tools become more accessible, and scientists face new obstacles to data collection (e.g. rising costs, declining survey response rates), researchers increasingly use predictions from pre-trained algorithms as outcome variables. Though appealing for financial and logistical reasons, using standard tools for inference can misrepresent the association between independent variables and the outcome of interest when the true, unobserved outcome is replaced by a predicted value. In this paper, we characterize the statistical challenges inherent to this so-called ``inference with predicted data'' problem and elucidate three potential sources of error: (i) the relationship between predicted outcomes and their true, unobserved counterparts, (ii) robustness of the machine learning model to resampling or uncertainty about the training data, and (iii) appropriately propagating not just bias but also uncertainty from predictions into the ultimate inference procedure.

Do We Really Even Need Data?

TL;DR

The statistical challenges inherent to this so-called ``inference with predicted data'' problem are characterized and three potential sources of error are elucidated, including the relationship between predicted outcomes and their true, unobserved counterparts.

Abstract

As artificial intelligence and machine learning tools become more accessible, and scientists face new obstacles to data collection (e.g. rising costs, declining survey response rates), researchers increasingly use predictions from pre-trained algorithms as outcome variables. Though appealing for financial and logistical reasons, using standard tools for inference can misrepresent the association between independent variables and the outcome of interest when the true, unobserved outcome is replaced by a predicted value. In this paper, we characterize the statistical challenges inherent to this so-called ``inference with predicted data'' problem and elucidate three potential sources of error: (i) the relationship between predicted outcomes and their true, unobserved counterparts, (ii) robustness of the machine learning model to resampling or uncertainty about the training data, and (iii) appropriately propagating not just bias but also uncertainty from predictions into the ultimate inference procedure.
Paper Structure (6 sections, 3 figures, 1 table)

This paper contains 6 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. The Problem
  3. Doesn't This Sound Familiar?
  4. Inference on Predicted Data and Current Salient Problems of Data Science
  5. Conclusions
  6. Acknowledgments

Figures (3)

  • Figure 1: Artist renderings of a rhinoceros based on limited information. Left: Albrecht Dürer's The Rhinoceros, woodcutting (1515); Right: C.M. Kösemen's re-imagining of a rhinoceros based on its skeleton.
  • Figure 2: Example general overview of inference on predicted data (IPD) based on the Prediction-Powered Inference (PPI) and PPI++ framework of Angelopoulos et al. (2023a, 2023b) angelopoulos2023predictionangelopoulos2023ppi++
  • Figure 3: Alternative methods for analyzing data with missing outcomes