Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal Data Integration and Adaptive Sampling for Efficient Treatment Effect Estimation

Yen-Chun Liu, Alexander Volfovsky, German Schnaidt, Cristobal Garib, Eric Laber

Abstract

This study addresses the challenge of estimating average treatment effects (ATEs) for advertising campaigns in online marketplaces where complete randomized experimentation is infeasible. We propose two key innovations: (1) a shrinkage estimator that optimally combines observational and experimental data without assuming smooth treatment effects across campaigns, and (2) a Bayesian adaptive experimental design framework that efficiently selects campaigns for randomized evaluation that minimizes cumulative risk. Our shrinkage estimator achieves lower risk compared to existing methods by balancing bias-variance tradeoffs, while our adaptive design significantly reduces the costs of campaign randomization. We establish theoretical guarantees including asymptotic normality and regret bounds. In an application to Amazon Ads data analyzing 2,583 campaigns, our approach achieves equivalent estimation precision while requiring only half of the randomized experiments needed by random sampling, the standard method widely used in practice today. The proposed method serves as a practical solution for marketplace platforms to efficiently measure advertising effectiveness while managing experimentation costs.

Optimal Data Integration and Adaptive Sampling for Efficient Treatment Effect Estimation

Abstract

This study addresses the challenge of estimating average treatment effects (ATEs) for advertising campaigns in online marketplaces where complete randomized experimentation is infeasible. We propose two key innovations: (1) a shrinkage estimator that optimally combines observational and experimental data without assuming smooth treatment effects across campaigns, and (2) a Bayesian adaptive experimental design framework that efficiently selects campaigns for randomized evaluation that minimizes cumulative risk. Our shrinkage estimator achieves lower risk compared to existing methods by balancing bias-variance tradeoffs, while our adaptive design significantly reduces the costs of campaign randomization. We establish theoretical guarantees including asymptotic normality and regret bounds. In an application to Amazon Ads data analyzing 2,583 campaigns, our approach achieves equivalent estimation precision while requiring only half of the randomized experiments needed by random sampling, the standard method widely used in practice today. The proposed method serves as a practical solution for marketplace platforms to efficiently measure advertising effectiveness while managing experimentation costs.

Paper Structure

This paper contains 15 sections, 68 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Notation and setup
  4. Estimation of intervention effects
  5. Proposed methods
  6. Optimal shrinkage estimator $\widehat{\pmb{\tau}}^\lambda_m$
  7. Bayesian Adaptive design
  8. Simulation Studies
  9. Application
  10. Discussion
  11. Proof of Lemmas and Theorem
  12. Proof:
  13. Proof:
  14. Proof:
  15. Consistency and asymptotic normality

Figures (3)

  • Figure 1: [Left] Risk of $\widehat{\pmb{\tau}}^\lambda$ using different shrinkage factors $\lambda\in[0,1]$ at round 1, 10, and 20. Dotted vertical lines indicate the estimated optimal shrinkage factor ($\widehat{\lambda}^{*}$); [Right] Cumulative risk (log scale) of various designs and sampling methods using the optimal shrinkage estimator.
  • Figure 2: An example of the advertising campaigns of interest, highlighted by the red box. Brands and merchants are fictional.
  • Figure 3: [Left] Estimated optimal shrinkage factors; [Right] Estimated RCT costs versus instantaneous risk. The risk difference is defined as the the difference between the risk of $\widehat{\pmb{\tau}}_m^{\lambda}(\overline{\mathbf{S}}_m)$ and the risk of the shrinkage estimator using RCT data of all 2,583 campaigns. Shaded areas represent one standard error. Dotted line indicates the risk using all RCT estimates at the final round.