Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Robust are Robustness Checks?

Brenda Prallon

Abstract

Robustness checks are routine in empirical work, but there is no standard statistical procedure to formally measure what one can learn from them. I propose a "robustness radius" measure to quantify the amount by which the robustness checks estimands differ from the main specification estimand. I do so by framing robustness checks as explicitly biased regressions, clarifying what exactly the estimands are when comparing multiple regressions with slightly different samples, and applying a test from the moment inequalities literature. The robustness radius is easily interpretable and adapts to sampling uncertainty and correlation across regressions. An application shows that, although assessing overall robustness is context-specific, the robustness radius guides those judgments and improves transparency.

How Robust are Robustness Checks?

Abstract

Robustness checks are routine in empirical work, but there is no standard statistical procedure to formally measure what one can learn from them. I propose a "robustness radius" measure to quantify the amount by which the robustness checks estimands differ from the main specification estimand. I do so by framing robustness checks as explicitly biased regressions, clarifying what exactly the estimands are when comparing multiple regressions with slightly different samples, and applying a test from the moment inequalities literature. The robustness radius is easily interpretable and adapts to sampling uncertainty and correlation across regressions. An application shows that, although assessing overall robustness is context-specific, the robustness radius guides those judgments and improves transparency.
Paper Structure (7 sections, 1 theorem, 33 equations, 3 figures, 5 tables)

This paper contains 7 sections, 1 theorem, 33 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Set-up
  3. Simulations and Illustrations
  4. Application: "Wage Stagnation and the Decline of Standardized Pay Rates, 1974–1991"
  5. Connection with Sensitivity Measures
  6. Conclusion
  7. Proof of Proposition \ref{['prop:RCC_test']}

Key Result

Proposition 1

Suppose Assumption assum:1 holds. Let $F \in \mathcal{F}$ denote the data distribution giving rise to $\hat{\theta}$ and $b(\theta) = \max_{j \in \{1, \ldots, m\}} |\theta_0 - \theta_j|$. Then, Theorems 1, 4 and 5 of coxandshi hold, and:

Figures (3)

  • Figure 1: Histogram of the robustness radius with different variance matrices. $\theta_0 = 0$, $\theta_1 = 1.5$, $\sigma_0 = 1$, $1{,}000$ simulations, $\alpha = 0.05$.
  • Figure 2: Average $b_{RR}$ following specifications in coxandshi.
  • Figure 3: Average $b_{RR}$ for different correlations $\rho$, and different $m$. $\sigma_j = 1$ for all $j$.

Theorems & Definitions (6)

  • Definition 1: Robustness radius
  • Proposition 1
  • proof
  • Definition 2: Full Robustness
  • Definition 3: Robust with respect to sign
  • proof