Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sample size planning for conditional counterfactual mean estimation with a K-armed randomized experiment

Gabriel Ruiz

TL;DR

Using policy trees to learn sub-groups and nominal guarantees on a large publicly-available randomized experiment test data set, the result allows us to invert the question to one about the feasible number of treatment arms or partition complexity (e.g. number of decision tree leaves).

Abstract

We cover how to determine a sufficiently large sample size for a $K$-armed randomized experiment in order to estimate conditional counterfactual expectations in data-driven subgroups. The sub-groups can be output by any feature space partitioning algorithm, including as defined by binning users having similar predictive scores or as defined by a learned policy tree. After carefully specifying the inference target, a minimum confidence level, and a maximum margin of error, the key is to turn the original goal into a simultaneous inference problem where the recommended sample size to offset an increased possibility of estimation error is directly related to the number of inferences to be conducted. Given a fixed sample size budget, our result allows us to invert the question to one about the feasible number of treatment arms or partition complexity (e.g. number of decision tree leaves). Using policy trees to learn sub-groups, we evaluate our nominal guarantees on a large publicly-available randomized experiment test data set.

Sample size planning for conditional counterfactual mean estimation with a K-armed randomized experiment

TL;DR

Using policy trees to learn sub-groups and nominal guarantees on a large publicly-available randomized experiment test data set, the result allows us to invert the question to one about the feasible number of treatment arms or partition complexity (e.g. number of decision tree leaves).

Abstract

We cover how to determine a sufficiently large sample size for a -armed randomized experiment in order to estimate conditional counterfactual expectations in data-driven subgroups. The sub-groups can be output by any feature space partitioning algorithm, including as defined by binning users having similar predictive scores or as defined by a learned policy tree. After carefully specifying the inference target, a minimum confidence level, and a maximum margin of error, the key is to turn the original goal into a simultaneous inference problem where the recommended sample size to offset an increased possibility of estimation error is directly related to the number of inferences to be conducted. Given a fixed sample size budget, our result allows us to invert the question to one about the feasible number of treatment arms or partition complexity (e.g. number of decision tree leaves). Using policy trees to learn sub-groups, we evaluate our nominal guarantees on a large publicly-available randomized experiment test data set.
Paper Structure (19 sections, 6 theorems, 60 equations, 3 figures)

This paper contains 19 sections, 6 theorems, 60 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Feature space partitioning model for conditional counterfactual means
  3. A manageable inference target in practice
  4. Main result
  5. Some regularity conditions
  6. The main result
  7. Applying Propositions \ref{['prop:CLTBased']}, \ref{['prop:boundedOutcome']}, and \ref{['prop:boundedOutcomeBoundedVar']} with a standardized scale
  8. Learning a feature space partition
  9. Given a fixed sample size
  10. Bounding the Outcome's conditional variance
  11. Emprical Evaluation
  12. Discussion and Limitations
  13. Acknowledgements
  14. Extended Main Results
  15. Proofs
  16. ...and 4 more sections

Key Result

Lemma 2.1

Let $\alpha\in(0,1)$ and $\epsilon>0$. Consider a new random test point $X_{n+1}$. If then along with Moreover, if then along with

Figures (3)

  • Figure 1: Example power analysis showing a sufficient sample size needed for jointly inferring the expected potential outcome in 3 treatment arms for a new randomly sampled individual. Here, individuals can fall into one of the sub-groups defined by a partition of the feature space given by an arbitrary decision tree. In addition to the number of sub-groups, the sample size is determined by a desired confidence level, margin of error, and unexplained outcome variation (or standard deviation).
  • Figure 2: Coverage across 500 replicates.
  • Figure 3: Across 500 replicates, the realized decision tree parameters.

Theorems & Definitions (11)

  • Lemma 2.1: Implications of accurate conditional counterfactual mean estimation
  • Proposition 2.1: Main Result: Central Limit Theorem
  • Remark 2.1: Working on a standardized scale
  • Proposition A.1: Main Result: Bounded Outcome
  • Proposition A.2: Main Result: Bounded Outcome and Bound on its Conditional Variance
  • proof : Proof of Lemma \ref{['lem:keyIneqs']}
  • proof : Proof of Equation \ref{['eqn:nonZeroVarBound']}
  • Lemma B.1: Joint estimation of independent means
  • proof
  • Lemma B.2: Joint estimation of independent group means nested within clusters
  • ...and 1 more