Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Balancing Covariates in Survey Experiments

Pengfei Tian, Jiyang Ren, Yingying Ma

TL;DR

This paper tackles covariate imbalance in population-based survey experiments by proposing a two-stage strategy—stratified rejective sampling and stratified rerandomization (SRSRR)—and a covariate-adjusted analysis. It develops a design-based asymptotic framework showing that SRSE yields consistency and asymptotic normality, while SRSRR leads to a convolution-type limiting distribution that concentrates more tightly around the true effect. The authors also derive conservative variance estimators and propose optimal allocation rules to minimize asymptotic variance, with regression adjustment further boosting efficiency. Numerical experiments with synthetic data and Cooperative Congressional Election Study (CCES) data demonstrate substantial efficiency gains from stratification, rerandomization, and covariate adjustment, supporting broad applicability for improving inference in survey experiments.

Abstract

The survey experiment is widely used in economics and social sciences to evaluate the effects of treatments or programs. In a standard population-based survey experiment, the experimenter randomly draws experimental units from a target population of interest and then randomly assigns the sampled units to treatment or control conditions to explore the treatment effect of an intervention. Simple random sampling and treatment assignment can balance covariates on average. However, covariate imbalance often exists in finite samples. To address the imbalance issue, we study a stratified approach to balance covariates in a survey experiment. A stratified rejective sampling and rerandomization design is further proposed to enhance the covariate balance. We develop a design-based asymptotic theory for the widely used stratified difference-in-means estimator of the average treatment effect under the proposed design. In particular, we show that it is consistent and asymptotically a convolution of a normal distribution and two truncated normal distributions. This limiting distribution is more concentrated at the true average treatment effect than that under the existing experimental designs. Moreover, we propose a covariate adjustment method in the analysis stage, which can further improve the estimation efficiency. Numerical studies demonstrate the validity and improved efficiency of the proposed method.

Balancing Covariates in Survey Experiments

TL;DR

This paper tackles covariate imbalance in population-based survey experiments by proposing a two-stage strategy—stratified rejective sampling and stratified rerandomization (SRSRR)—and a covariate-adjusted analysis. It develops a design-based asymptotic framework showing that SRSE yields consistency and asymptotic normality, while SRSRR leads to a convolution-type limiting distribution that concentrates more tightly around the true effect. The authors also derive conservative variance estimators and propose optimal allocation rules to minimize asymptotic variance, with regression adjustment further boosting efficiency. Numerical experiments with synthetic data and Cooperative Congressional Election Study (CCES) data demonstrate substantial efficiency gains from stratification, rerandomization, and covariate adjustment, supporting broad applicability for improving inference in survey experiments.

Abstract

The survey experiment is widely used in economics and social sciences to evaluate the effects of treatments or programs. In a standard population-based survey experiment, the experimenter randomly draws experimental units from a target population of interest and then randomly assigns the sampled units to treatment or control conditions to explore the treatment effect of an intervention. Simple random sampling and treatment assignment can balance covariates on average. However, covariate imbalance often exists in finite samples. To address the imbalance issue, we study a stratified approach to balance covariates in a survey experiment. A stratified rejective sampling and rerandomization design is further proposed to enhance the covariate balance. We develop a design-based asymptotic theory for the widely used stratified difference-in-means estimator of the average treatment effect under the proposed design. In particular, we show that it is consistent and asymptotically a convolution of a normal distribution and two truncated normal distributions. This limiting distribution is more concentrated at the true average treatment effect than that under the existing experimental designs. Moreover, we propose a covariate adjustment method in the analysis stage, which can further improve the estimation efficiency. Numerical studies demonstrate the validity and improved efficiency of the proposed method.
Paper Structure (35 sections, 18 theorems, 150 equations, 6 tables)

This paper contains 35 sections, 18 theorems, 150 equations, 6 tables.

Table of Contents

  1. Introduction
  2. Stratified rejective sampling and rerandomization
  3. Stratified population-based survey experiment
  4. Stratified rejective sampling and rerandomization
  5. Design-based asymptotic theory
  6. Asymptotic properties under SRSE
  7. Asymptotic properties under SRSRR
  8. Conservative variance and distribution estimators
  9. Regression adjustment
  10. Numerical studies
  11. Synthetic data
  12. Cooperative congressional election study data
  13. Discussion
  14. Additional theoretical results
  15. Mean and covariance of $\sqrt{n}(\hat{\tau} -\tau,\hat{\tau}_X^\mathrm{T},\hat{\delta}_W^\mathrm{T})^\mathrm{T}$
  16. ...and 20 more sections

Key Result

Theorem 1

Under Condition cond srse and the stratified randomized survey experiment, we have $\sqrt n(\hat{\tau} -\tau,\hat{\tau}_X^\mathrm{T},\hat{\delta}_W^\mathrm{T})^\mathrm{T} \xrightarrow{d} \mathcal{N}(0, V).$

Theorems & Definitions (22)

  • Theorem 1
  • Corollary 1
  • Remark 1
  • Theorem 2
  • Theorem 3
  • Remark 2
  • Corollary 2
  • Theorem 4
  • Remark 3
  • Theorem 5
  • ...and 12 more