An Axiomatic Approach to Comparing Sensitivity Parameters
Paul Diegert, Matthew A. Masten, Alexandre Poirier
TL;DR
This paper addresses how to choose among sensitivity analyses for omitted-variable bias in linear regression by introducing a formal, design-based framework based on covariate sampling. It defines equal selection as a benchmark and two key properties—consistency and monotonicity in selection—to evaluate sensitivity parameters, showing that Oster2019’s delta is inconsistent and non-monotonic while CinelliHazlett2020 and DMP2023v5 satisfy both. Through high-level asymptotics and lower-level covariance structures (MA, AR, and factor models), it characterizes when common parameters converge to interpretable benchmarks (e.g., r_X(S) to $\sqrt{\frac{r(r+c_\pi)}{1+r c_\pi}}$ and delta_ACET to 1) and identifies parameters that fail these properties (e.g., delta_resid). An empirical exercise using BFG2020 data corroborates the theoretical findings and demonstrates that some parameters align with intuitive robustness benchmarks while others do not, guiding applied researchers toward more reliable sensitivity analyses. Overall, the framework provides a principled basis to compare and select sensitivity measures, with practical implications for robustness checks beyond the linear-effects context and concrete guidance favoring consistency- and monotonicity-compliant methods.
Abstract
Many methods are available for assessing the importance of omitted variables in linear regression. These methods typically make different, non-falsifiable assumptions. Hence the data alone cannot tell us which method is most appropriate. Since it is unreasonable to expect results to be robust against all possible robustness checks, researchers often use methods deemed ``interpretable,'' a subjective criterion with no formal definition. In contrast, we develop the first formal, axiomatic framework for comparing and selecting among these methods. Our framework is analogous to the standard approach for comparing estimators based on their sampling distributions. We propose that sensitivity parameters be selected based on their covariate sampling distributions, a design distribution of parameter values induced by an assumption on how covariates are assigned to be observed or unobserved. Using this idea, we define new concepts of parameter consistency and monotonicity, and argue that a reasonable sensitivity parameter should satisfy both properties. We prove that the literature's most popular approach is inconsistent and non-monotonic, while several alternatives satisfy both.