Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal Sample Splitting for Observational Studies

Qishuo Yin, Dylan S. Small

Abstract

In observational studies of treatment effects, estimates may be biased by unmeasured confounders, which can potentially affect the validity of the results. Understanding sensitivity to such biases helps assess how unmeasured confounding impacts credibility. The design of an observational study strongly influences its sensitivity to bias. Previous work has shown that the sensitivity to bias can be reduced by dividing a dataset into a planning sample and a larger analysis sample, where the planning sample guides design decisions. But the choice of what fraction of the data to put in the planning sample vs. the analysis sample was ad hoc. Here, we develop an approach to find the optimal fraction using plasmode datasets. We show that our method works well in high-dimensional outcome spaces. We apply our method to study the effects of exposure to second-hand smoke in children. The OptimalSampling R package implementing our method is available at GitHub.

Optimal Sample Splitting for Observational Studies

Abstract

In observational studies of treatment effects, estimates may be biased by unmeasured confounders, which can potentially affect the validity of the results. Understanding sensitivity to such biases helps assess how unmeasured confounding impacts credibility. The design of an observational study strongly influences its sensitivity to bias. Previous work has shown that the sensitivity to bias can be reduced by dividing a dataset into a planning sample and a larger analysis sample, where the planning sample guides design decisions. But the choice of what fraction of the data to put in the planning sample vs. the analysis sample was ad hoc. Here, we develop an approach to find the optimal fraction using plasmode datasets. We show that our method works well in high-dimensional outcome spaces. We apply our method to study the effects of exposure to second-hand smoke in children. The OptimalSampling R package implementing our method is available at GitHub.
Paper Structure (26 sections, 22 equations, 2 figures, 6 tables)

This paper contains 26 sections, 22 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Notation and Review
  3. Paired Randomized Experiments
  4. Sensitivity Analysis
  5. Design Evaluation
  6. Method
  7. Background
  8. Motivation and Contribution
  9. Optimal Sample Split Multiple Tests
  10. Implementation Details
  11. Example: Wilcoxon’s Signed Rank Statistic
  12. Asymptotic Properties of Wilcoxon's Signed Rank Statistic
  13. Simulation
  14. Simulation Methods
  15. Data Generating Process
  16. ...and 11 more sections

Figures (2)

  • Figure 1: Test power vs. sample split fraction under multiple hypothesis tests for outcomes $K = 10$, sample size $I = 200$, and bias factor $\Gamma = 1$ by selection with $1000$ replications.
  • Figure 2: Box plot for the matched nonbinary covariates - age, poverty, number of rooms in the house, and size of the family - between treated and controlled groups.