Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects

Santiago Cortes-Gomez, Naveen Raman, Aarti Singh, Bryan Wilder

TL;DR

This work develops a two-stage RCT where, first on a data-driven screening stage, it prune low-impact treatments, while in the second stage, it develops high probability lower bounds on the treatment effect.

Abstract

Randomized controlled trials (RCTs) can be used to generate guarantees on treatment effects. However, RCTs often spend unnecessary resources exploring sub-optimal treatments, which can reduce the power of treatment guarantees. To address these concerns, we develop a two-stage RCT where, first on a data-driven screening stage, we prune low-impact treatments, while in the second stage, we develop high probability lower bounds on the treatment effect. Unlike existing adaptive RCT frameworks, our method is simple enough to be implemented in scenarios with limited adaptivity. We derive optimal designs for two-stage RCTs and demonstrate how we can implement such designs through sample splitting. Empirically, we demonstrate that two-stage designs improve upon single-stage approaches, especially in scenarios where domain knowledge is available in the form of a prior. Our work is thus, a simple, yet effective, method to estimate high probablility certificates for high performant treatment effects on a RCT.

Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects

TL;DR

This work develops a two-stage RCT where, first on a data-driven screening stage, it prune low-impact treatments, while in the second stage, it develops high probability lower bounds on the treatment effect.

Abstract

Randomized controlled trials (RCTs) can be used to generate guarantees on treatment effects. However, RCTs often spend unnecessary resources exploring sub-optimal treatments, which can reduce the power of treatment guarantees. To address these concerns, we develop a two-stage RCT where, first on a data-driven screening stage, we prune low-impact treatments, while in the second stage, we develop high probability lower bounds on the treatment effect. Unlike existing adaptive RCT frameworks, our method is simple enough to be implemented in scenarios with limited adaptivity. We derive optimal designs for two-stage RCTs and demonstrate how we can implement such designs through sample splitting. Empirically, we demonstrate that two-stage designs improve upon single-stage approaches, especially in scenarios where domain knowledge is available in the form of a prior. Our work is thus, a simple, yet effective, method to estimate high probablility certificates for high performant treatment effects on a RCT.

Paper Structure

This paper contains 17 sections, 5 theorems, 7 equations, 7 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Introducing Two-Stage RCTs
  4. Problem Formulation
  5. Certificate estimation
  6. Finding optimal policies for Two-Stage RCTs
  7. Designing Top-K Policies
  8. Sample Splitting
  9. Incorporating Priors
  10. Experiments
  11. Synthetic Dataset and Setup
  12. Comparison against Single-Stage Designs
  13. Impact of Budget
  14. Comparison with Adaptive Policies
  15. Bayesian Setting
  16. ...and 2 more sections

Key Result

Lemma 0

Let $\sigma$ be the descending ordering of arms by empirical mean observed on the first stage. Then, for any $i < j$, $D_{\mu_{\sigma_i(\mathbf{X})}}$ first-order stochastically dominates, $D_{\mu_{\sigma_i(\mathbf{X})}}$

Figures (7)

  • Figure 1: Our sample splitting algorithm outperforms all baselines across first stage sizes. The largest improvement occurs when the first stage is small, as this leaves budget for the second stage to compute certificates.
  • Figure 2: Sample split algorithms perform well for all values of $T$. When $T$ is large, the sample split approaches the optimal two-stage policy (omniscient).
  • Figure 3: Single-stage methods perform best when between 20% and 70% of the budget spent in the first stage, as this allows for arms to be pruned, and a certificate to be generated in the second stage.
  • Figure 4: Sample splitting policies can close the gap between single-stage and adaptive policies (such as UCB), and serves as a middle ground in performance and complexity. This is best seen when the first stage is 30%, as sample split methods can capture up to 69% of the improvement between single-stage and UCB designs.
  • Figure 5: When priors are informative, correlating to large $\beta$, prior-based methods can improve upon all policies, including adaptive policies such as UCB and two-stage Thompson Sampling.
  • ...and 2 more figures

Theorems & Definitions (8)

  • Definition 1
  • Definition 2
  • Definition 3
  • Lemma 0
  • Lemma 0
  • Theorem 1
  • Proposition 1
  • Theorem 2