Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Automatic Finite-Sample Robustness Metric: When Can Dropping a Little Data Make a Big Difference?

Tamara Broderick, Ryan Giordano, Rachael Meager

TL;DR

The paper tackles the problem that empirical econometric conclusions can be highly sensitive to small nonrandom departures from the sample by introducing AMIP, a finite-sample robustness metric based on the empirical influence function. It develops a fast, Taylor-series-based approximation to gauge the worst-case impact of dropping data, provides an exact lower-bound refinement, and shows how to compute influence scores efficiently; theory is complemented by extensive applied experiments across health, cash transfer, and microcredit RCTs. Key findings show robustness is governed by the signal-to-noise ratio and that, in several cases, removing less than 1% of observations can reverse conclusions, even when standard errors suggest precision. The work argues AMIP should complement, not replace, existing robustness analyses, and provides practical tools and intuition for researchers to assess and report data-sensitivity in policy-relevant econometrics.

Abstract

Study samples often differ from the target populations of inference and policy decisions in non-random ways. Researchers typically believe that such departures from random sampling -- due to changes in the population over time and space, or difficulties in sampling truly randomly -- are small, and their corresponding impact on the inference should be small as well. We might therefore be concerned if the conclusions of our studies are excessively sensitive to a very small proportion of our sample data. We propose a method to assess the sensitivity of applied econometric conclusions to the removal of a small fraction of the sample. Manually checking the influence of all possible small subsets is computationally infeasible, so we use an approximation to find the most influential subset. Our metric, the "Approximate Maximum Influence Perturbation," is based on the classical influence function, and is automatically computable for common methods including (but not limited to) OLS, IV, MLE, GMM, and variational Bayes. We provide finite-sample error bounds on approximation performance. At minimal extra cost, we provide an exact finite-sample lower bound on sensitivity. We find that sensitivity is driven by a signal-to-noise ratio in the inference problem, is not reflected in standard errors, does not disappear asymptotically, and is not due to misspecification. While some empirical applications are robust, results of several influential economics papers can be overturned by removing less than 1% of the sample.

An Automatic Finite-Sample Robustness Metric: When Can Dropping a Little Data Make a Big Difference?

TL;DR

The paper tackles the problem that empirical econometric conclusions can be highly sensitive to small nonrandom departures from the sample by introducing AMIP, a finite-sample robustness metric based on the empirical influence function. It develops a fast, Taylor-series-based approximation to gauge the worst-case impact of dropping data, provides an exact lower-bound refinement, and shows how to compute influence scores efficiently; theory is complemented by extensive applied experiments across health, cash transfer, and microcredit RCTs. Key findings show robustness is governed by the signal-to-noise ratio and that, in several cases, removing less than 1% of observations can reverse conclusions, even when standard errors suggest precision. The work argues AMIP should complement, not replace, existing robustness analyses, and provides practical tools and intuition for researchers to assess and report data-sensitivity in policy-relevant econometrics.

Abstract

Study samples often differ from the target populations of inference and policy decisions in non-random ways. Researchers typically believe that such departures from random sampling -- due to changes in the population over time and space, or difficulties in sampling truly randomly -- are small, and their corresponding impact on the inference should be small as well. We might therefore be concerned if the conclusions of our studies are excessively sensitive to a very small proportion of our sample data. We propose a method to assess the sensitivity of applied econometric conclusions to the removal of a small fraction of the sample. Manually checking the influence of all possible small subsets is computationally infeasible, so we use an approximation to find the most influential subset. Our metric, the "Approximate Maximum Influence Perturbation," is based on the classical influence function, and is automatically computable for common methods including (but not limited to) OLS, IV, MLE, GMM, and variational Bayes. We provide finite-sample error bounds on approximation performance. At minimal extra cost, we provide an exact finite-sample lower bound on sensitivity. We find that sensitivity is driven by a signal-to-noise ratio in the inference problem, is not reflected in standard errors, does not disappear asymptotically, and is not due to misspecification. While some empirical applications are robust, results of several influential economics papers can be overturned by removing less than 1% of the sample.

Paper Structure

This paper contains 36 sections, 4 theorems, 62 equations, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. A proposed measure of sensitivity to dropping small data subsets
  3. A Taylor series approximation to dropping data
  4. A tractable approximation
  5. An exact lower bound on the Maximum Influence Perturbation
  6. Computing the influence scores
  7. Example functions of interest
  8. A real-world OLS regression example
  9. Underlying theory and interpretation
  10. Theory and interpretation for Ordinary Least Squares
  11. Problem setup for Ordinary Least Squares example
  12. What determines AMIP robustness for Ordinary Least Squares?
  13. Theory and interpretation for general Z-estimators
  14. The decomposition of the AMIP
  15. What determines AMIP robustness?
  16. ...and 21 more sections

Key Result

Theorem 1

thetafun_accuracy Let ij_assu, thetafun_smooth hold. For sufficiently small $\alpha$, there exist constants $C_1$ and $C_2$, defined in terms of quantities given in ij_assu, thetafun_smooth, such thatWe note that the rate $\sqrt{\alpha}$ is determined by a simple but coarse Cauchy-Schwartz bound (se

Figures (1)

  • Figure 1: Simulation results for univariate linear regression with $N = 5,000$ observations. Left panel: The approximate perturbation inducing proportion at differing values of $\sigma_x$ and $\sigma_\varepsilon$. Red colors indicate datasets whose sign can is predicted to change when dropping less than 1% of datapoints. The grey areas indicate $\hat{\Psi}_{\alpha} = \texttt{NA}$, a failure of the linear approximation to locate any way to change the sign. Right panel: The actual change, linear approximation to the change, and approximation error for $\sigma_x = 2$ and $\sigma_\varepsilon = 1$.

Theorems & Definitions (9)

  • Definition 1
  • Definition 2
  • Theorem 1
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof