Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sequentially-Rerandomized Switchback Experiments

Zhenghao Zeng, Christopher Adjaho, Alonso Bucarey, Chao Qin, Ruixuan Zhang, Paul Hoban, Ramesh Johari, Stefan Wager

Abstract

Large-scale online platforms and marketplace systems often evaluate new policies through experiments that randomize treatment across operational units (e.g., geographies, regions, or clusters) over many time periods. In these settings, standard A/B testing can be inefficient or unreliable due to a limited number of units, substantial cross-unit heterogeneity, non-stationarity, and potential carryover across periods. We propose Sequentially-Rerandomized Switchback Experiments (SRSB), a new experimental design that helps mitigate these challenges. SRSB re-randomizes treatment at each time period such as to enforce balance on pre-specified prognostic variables constructed from past observations. In the absence of carryover, SRSB improves precision by leveraging temporal dependence through balancing lagged outcomes and covariates; we develop finite-sample randomization inference under a sharp null as well as asymptotic inference as the number of periods grows. We then extend SRSB to settings with first-order carryover and introduce a blocked SRSB variant that rerandomizes within strata defined by the previous treatment to form stable and comparable "stay" groups. Extensive simulations demonstrate the practical gains and robustness of SRSB relative to standard switchback designs.

Sequentially-Rerandomized Switchback Experiments

Abstract

Large-scale online platforms and marketplace systems often evaluate new policies through experiments that randomize treatment across operational units (e.g., geographies, regions, or clusters) over many time periods. In these settings, standard A/B testing can be inefficient or unreliable due to a limited number of units, substantial cross-unit heterogeneity, non-stationarity, and potential carryover across periods. We propose Sequentially-Rerandomized Switchback Experiments (SRSB), a new experimental design that helps mitigate these challenges. SRSB re-randomizes treatment at each time period such as to enforce balance on pre-specified prognostic variables constructed from past observations. In the absence of carryover, SRSB improves precision by leveraging temporal dependence through balancing lagged outcomes and covariates; we develop finite-sample randomization inference under a sharp null as well as asymptotic inference as the number of periods grows. We then extend SRSB to settings with first-order carryover and introduce a blocked SRSB variant that rerandomizes within strata defined by the previous treatment to form stable and comparable "stay" groups. Extensive simulations demonstrate the practical gains and robustness of SRSB relative to standard switchback designs.

Paper Structure

This paper contains 26 sections, 3 theorems, 108 equations, 10 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Setup & Assumptions
  3. Multi-unit Switchback Experiments
  4. Sequentially-Rerandomized Switchback Experiments
  5. SRSB Without Carryover Effects
  6. Sequential Rerandomization
  7. Estimation and Inference
  8. Randomization Inference
  9. Asymptotic Inference
  10. First-order Carryover effects
  11. Blocked SRSB Design
  12. Estimation and Inference
  13. Simulation Studies
  14. No Carryover Effects
  15. Carryover Effects
  16. ...and 11 more sections

Key Result

Theorem 1

Suppose the Lindeberg condition holds: For any $\epsilon >0$, and $V_T^2/S_T^2 \stackrel{P}{\rightarrow} 1$, then we have A sufficient condition for Lindeberg's condition is that there exists a constant $C>0$ such that $|Y_{i,t}| \leq C$ for all $i \in [N],t \in [T]$ and $S_T^2 \rightarrow \infty$. $\blacktriangleleft$$\blacktriangleleft$

Figures (10)

  • Figure 1: Results from a toy simulation example where the potential outcomes are generated from an AR(1) model with covariates; see Section \ref{['sec:simu-simple-nocarryover']} for details of the data-generating process. Top: Examples of average outcome trajectories under complete randomization at each time period. Bottom: Examples of average outcome trajectories under rerandomization that balances all previously observed outcomes in the same setting. The true treatment effect is $1$. At each time point, average outcomes are computed separately for groups defined by the treatment assignment at time $t=1$. The vertical dashed line at $t=0$ marks the decision time: outcomes up to $t=0$ are observed before assigning treatment at $t=1$. In the top panel, the difference-in-means estimator at $t=1$ can substantially overestimate (first two examples) or underestimate (third example) the treatment effect due to imbalance in lagged outcomes and pre-treatment trends. In the bottom panel, rerandomization reduces such imbalance by enforcing balance on lagged outcomes, leading to more accurate estimation of the treatment effect at $t=1$.
  • Figure 2: RMSE comparison between completely randomized experiments and sequential rerandomization experiments in the no-carryover setting. The potential outcomes are generated from an AR(1) model with covariates; see Section \ref{['sec:simu-simple-nocarryover']} for details of the data-generating process.
  • Figure 3: RMSE vs $\rho$ and proportion of RMSE reduced by sequential rerandomization. The potential outcomes are generated from an AR(1) model with covariates; see Section \ref{['sec:simu-simple-nocarryover']} for details of the data-generating process.
  • Figure 4: Comparison of RMSE under different experimental designs in a setting with first-order carryover effects. The potential outcomes are generated from an AR(1) model with covariates; see Section \ref{['sec:simu-carryover']} for details of the data-generating process.
  • Figure 5: Comparison of RMSE across different treatment effect sizes. The potential outcomes are generated from a factor model fitted to the Penn World Table; see Section \ref{['sec:simu-gdp']} for details.
  • ...and 5 more figures

Theorems & Definitions (7)

  • Theorem 1
  • Theorem 2
  • proof
  • proof
  • Lemma 1
  • proof
  • Definition 1: Mixingales