Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Randomization Tests in Switchback Experiments

Jizhou Liu, Liang Zhong

Abstract

Switchback experiments--alternating treatment and control over time--are widely used when unit-level randomization is infeasible, outcomes are aggregated, or user interference is unavoidable. In practice, experimentation must support fast product cycles, so teams often run studies for limited durations and make decisions with modest samples. At the same time, outcomes in these time-indexed settings exhibit serial dependence, seasonality, and occasional heavy-tailed shocks, and temporal interference (carryover or anticipation) can render standard asymptotics and naive randomization tests unreliable. In this paper, we develop a randomization-test framework that delivers finite-sample valid, distribution-free p-values for several null hypotheses of interest using only the known assignment mechanism, without parametric assumptions on the outcome process. For causal effects of interests, we impose two primitive conditions--non-anticipation and a finite carryover horizon m--and construct conditional randomization tests (CRTs) based on an ex ante pooling of design blocks into "sections," which yields a tractable conditional assignment law and ensures imputability of focal outcomes. We provide diagnostics for learning the carryover window and assessing non-anticipation, and we introduce studentized CRTs for a session-wise weak null that accommodates within-session seasonality with asymptotic validity. Power approximations under distributed-lag effects with AR(1) noise guide design and analysis choices, and simulations demonstrate favorable size and power relative to common alternatives. Our framework extends naturally to other time-indexed designs.

Randomization Tests in Switchback Experiments

Abstract

Switchback experiments--alternating treatment and control over time--are widely used when unit-level randomization is infeasible, outcomes are aggregated, or user interference is unavoidable. In practice, experimentation must support fast product cycles, so teams often run studies for limited durations and make decisions with modest samples. At the same time, outcomes in these time-indexed settings exhibit serial dependence, seasonality, and occasional heavy-tailed shocks, and temporal interference (carryover or anticipation) can render standard asymptotics and naive randomization tests unreliable. In this paper, we develop a randomization-test framework that delivers finite-sample valid, distribution-free p-values for several null hypotheses of interest using only the known assignment mechanism, without parametric assumptions on the outcome process. For causal effects of interests, we impose two primitive conditions--non-anticipation and a finite carryover horizon m--and construct conditional randomization tests (CRTs) based on an ex ante pooling of design blocks into "sections," which yields a tractable conditional assignment law and ensures imputability of focal outcomes. We provide diagnostics for learning the carryover window and assessing non-anticipation, and we introduce studentized CRTs for a session-wise weak null that accommodates within-session seasonality with asymptotic validity. Power approximations under distributed-lag effects with AR(1) noise guide design and analysis choices, and simulations demonstrate favorable size and power relative to common alternatives. Our framework extends naturally to other time-indexed designs.
Paper Structure (81 sections, 23 theorems, 270 equations, 3 figures, 3 algorithms)

This paper contains 81 sections, 23 theorems, 270 equations, 3 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Setup and Notation
  3. Switchback experiments and regular block randomization
  4. Assumptions on temporal interference
  5. Main Results
  6. Overview of Conditional Randomization Tests
  7. Testing total treatment effects
  8. Testing the carryover horizon
  9. Testing non-anticipation
  10. Imputable times.
  11. Prefix-preserving randomization.
  12. Centered signed-score statistic.
  13. PIRT procedure.
  14. Finite-sample validity.
  15. Testing Weak-null Hypotheses
  16. ...and 66 more sections

Key Result

Lemma 3.1

Suppose blocks $W^{(k)}$ are independent with $W^{(k)} \sim \mathrm{Bern}(q_k)$. For a merged section covering blocks $a,\ldots,b$, conditional on the event $\{W^{(a)} = \cdots = W^{(b)}\}$, the common value is Bernoulli with probability Moreover, across disjoint sections these conditional draws are independent.

Figures (3)

  • Figure 1: Rejection Frequencies Under the Null of no total treatment effect
  • Figure 2: Rejection Frequencies Under the Carryover Null and Alternatives
  • Figure 3: Rejection Frequencies Under the Anticipation Null and Alternatives

Theorems & Definitions (50)

  • Definition 2.1: Regular switchback experiment
  • Remark 2.1: Interpretation of $m$
  • Lemma 3.1: Conditional law on a constant merged section
  • Theorem 3.1: Finite-sample validity
  • Remark 3.1: Predetermined sections and invariance
  • Remark 3.2: Permutation implementations under homogeneous section probabilities
  • Definition 3.1: Null of $m$-carryover effects $H_0^m$
  • Lemma 3.2: Local dependence of $Y_t$ under $H_0^m$
  • Proposition 3.1: Nestedness
  • Definition 3.2: FWER under nested nulls
  • ...and 40 more