Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Use of Design-Based Simulations

Bruno Ferman

Abstract

Design-based simulations - procedures that hold realized outcomes fixed and generate variation by resampling treatment assignment or shocks - are widely used in both methodological and applied work to assess inference procedures. This paper studies the extent to which such simulations are informative about inference validity. Focusing on shift-share designs, we show that standard simulations that fix outcomes and resample shocks may rely on a data-generating process that is not aligned with the true one. In particular, these simulations confound true treatment effects with error dependence, potentially overstating inference distortions due to spatial correlation. We propose alternative simulation designs that circumvent this problem and illustrate their use in prominent empirical applications. Our results highlight that the usefulness of design-based simulations depends critically on how closely the simulated data-generating process aligns with the true one.

On the Use of Design-Based Simulations

Abstract

Design-based simulations - procedures that hold realized outcomes fixed and generate variation by resampling treatment assignment or shocks - are widely used in both methodological and applied work to assess inference procedures. This paper studies the extent to which such simulations are informative about inference validity. Focusing on shift-share designs, we show that standard simulations that fix outcomes and resample shocks may rely on a data-generating process that is not aligned with the true one. In particular, these simulations confound true treatment effects with error dependence, potentially overstating inference distortions due to spatial correlation. We propose alternative simulation designs that circumvent this problem and illustrate their use in prominent empirical applications. Our results highlight that the usefulness of design-based simulations depends critically on how closely the simulated data-generating process aligns with the true one.
Paper Structure (15 sections, 2 theorems, 28 equations, 1 figure, 3 tables)

This paper contains 15 sections, 2 theorems, 28 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Design-based simulations in randomized experiments
  3. Design-based simulations in shift-share designs
  4. Shift-share design framework
  5. Design-based simulations in shift-share designs: theory
  6. Simulations
  7. Empirical illustrations
  8. Analyzing distortions due to spatial correlation
  9. Assessing the reliability of asymptotic approximations
  10. Choosing among alternative inference methods
  11. Conclusion
  12. Appendix
  13. Proof of Proposition \ref{['Prop']}
  14. Simulations
  15. Probability of flagging spatial correlation

Key Result

Proposition 3.1

Consider the shift-share design setting described in this section, and assume the vectors $\{\epsilon_i: i \in \Lambda_f\}$ are iid across $f$ with $\mathbb{E}[\epsilon_i]=0$, $\mathbb{V}(\epsilon_i)=\sigma^2$, and $cov(\epsilon_i,\epsilon_s)=\rho$ for $i \neq s$ and $i,s \in \Lambda_f$ for some $f$

Figures (1)

  • Figure A.1: Probability of detecting spatial correlation problems

Theorems & Definitions (8)

  • Remark 1
  • Proposition 3.1
  • proof
  • Remark 2
  • Remark 3
  • proof
  • Proposition A.1
  • proof