Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finite Population Identification and Design-Based Sensitivity Analysis

Brendan Kline, Matthew A. Masten

Abstract

We develop a new approach for quantifying uncertainty in finite populations, by using design distributions to calibrate sensitivity parameters in finite population identified sets. This yields uncertainty intervals that can be interpreted as identified sets, robust Bayesian credible sets, or uniform frequentist design-based confidence sets. We focus on quantifying uncertainty about the average treatment effect, where our approach (1) yields design-based confidence intervals which allow for heterogeneous treatment effects without using asymptotics, (2) provides a new motivation for examining covariate balance, and (3) gives a new formal analysis of the role of randomization. We illustrate our approach in three empirical applications.

Finite Population Identification and Design-Based Sensitivity Analysis

Abstract

We develop a new approach for quantifying uncertainty in finite populations, by using design distributions to calibrate sensitivity parameters in finite population identified sets. This yields uncertainty intervals that can be interpreted as identified sets, robust Bayesian credible sets, or uniform frequentist design-based confidence sets. We focus on quantifying uncertainty about the average treatment effect, where our approach (1) yields design-based confidence intervals which allow for heterogeneous treatment effects without using asymptotics, (2) provides a new motivation for examining covariate balance, and (3) gives a new formal analysis of the role of randomization. We illustrate our approach in three empirical applications.

Paper Structure

This paper contains 32 sections, 10 theorems, 28 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Our Approach to Quantifying Uncertainty in Experiments: A Practitioner's Guide
  3. Related Literature
  4. Finite Population Identification
  5. The Finite Population Identified Set
  6. Identified Sets for ATE
  7. Randomization and Identification in Finite Populations
  8. The Prior Definition of Finite Population Identification
  9. Design-Based Sensitivity Analysis
  10. A Design-Based Approach to Calibrating $K$
  11. Interpreting $\Theta_I(K(\alpha))$
  12. The Value of Randomization
  13. Measuring the Strength of Evidence: $\underline{p}(K^\text{bp})$ as an Alternative to $p$-values
  14. Computing Bounds on the Design-Probability of $K$-Approximate Balance
  15. Numerical Illustration
  16. ...and 17 more sections

Key Result

Theorem 1

Suppose assump:boundedSupport1 and assump:KapproxBalance hold, and $\mathbf{P}^\text{data}$ is known. Then, for each $x \in \{0,1\}$, the identified set for $\overline{\mathbf{Y}(x)}$ is $[\text{LB}_K(x), \text{UB}_K(x)]$ where Moreover, the identified set for ATE is $\Theta_I(K) \coloneqq [\text{LB}_K(1) - \text{UB}_K(0), \; \text{UB}_K(1) - \text{LB}_K(0)]$.

Figures (4)

  • Figure 1: Example output for a design-based sensitivity analysis, $N=10$. See section \ref{['sec:Gneezy']} for data details.
  • Figure 2: Illustration of the convergence results of theorem \ref{['thm:randomizationGood']}.
  • Figure 3: Assessing accuracy of the genetic algorithm: $\underline{p}$ vs $K$ (left plot), $\Theta_I(K(\alpha))$ vs $1-\alpha$ (right plot) for GA (solid line) and MILP (dashed line), $N=10$.
  • Figure 4: Quantifying the uncertainty in the impact of management interventions on long run adoption of management practices.

Theorems & Definitions (22)

  • Definition 1
  • Theorem 1
  • Theorem 2
  • Definition 2: Conventional
  • Proposition 1
  • Proposition 2
  • proof : Proof of proposition \ref{['prop:weirdWeird']}
  • Proposition 3
  • Theorem 3
  • Theorem 4
  • ...and 12 more