Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable Decisions using a Bayesian Decision-Theoretic Approach

Hoiyi Ng, Guido Imbens

TL;DR

This work replaces traditional NHST in-production decision making with a Bayesian decision-theoretic framework that jointly accounts for multiple performance metrics through a linear loss and a decision rule that minimizes expected risk. It integrates hierarchical priors to borrow strength from historical experiments, enabling more efficient estimation and coherent, automated launch/rollback recommendations that reflect business trade-offs and costs. Through 70 real and 20 simulated supply-chain experiments, the approach shows substantial estimation efficiency gains, increased ability to detect true effects, and robust decision guidance under varying risk appetites. The paper also discusses practical loss-definition strategies and outlines extensions such as sequential testing and richer hierarchical evolution to further accelerate Amazon’s experimentation cadence.

Abstract

Randomized controlled experiments assess new policy impacts on performance metrics to inform launch decisions. Traditional approaches evaluate metrics independently despite correlations, and mixed results (e.g., positive revenue impact, negative customer experience) require manual judgment, hindering scalability. We propose a Bayesian decision-theoretic framework that systematically incorporates multiple objectives and trade-offs by comparing expected risks across decisions. Our approach combines experimenter-defined loss functions with observed evidence, using hierarchical models to leverage historical experiment learnings for prior information on treatment effects. Through real and simulated Amazon supply chain experiments, we demonstrate that compared to null hypothesis statistical testing, our method increases estimation efficiency via informative hierarchical priors and simplifies decision-making by systematically incorporating business preferences and costs for comprehensive, scalable decisions.

Scalable Decisions using a Bayesian Decision-Theoretic Approach

TL;DR

This work replaces traditional NHST in-production decision making with a Bayesian decision-theoretic framework that jointly accounts for multiple performance metrics through a linear loss and a decision rule that minimizes expected risk. It integrates hierarchical priors to borrow strength from historical experiments, enabling more efficient estimation and coherent, automated launch/rollback recommendations that reflect business trade-offs and costs. Through 70 real and 20 simulated supply-chain experiments, the approach shows substantial estimation efficiency gains, increased ability to detect true effects, and robust decision guidance under varying risk appetites. The paper also discusses practical loss-definition strategies and outlines extensions such as sequential testing and richer hierarchical evolution to further accelerate Amazon’s experimentation cadence.

Abstract

Randomized controlled experiments assess new policy impacts on performance metrics to inform launch decisions. Traditional approaches evaluate metrics independently despite correlations, and mixed results (e.g., positive revenue impact, negative customer experience) require manual judgment, hindering scalability. We propose a Bayesian decision-theoretic framework that systematically incorporates multiple objectives and trade-offs by comparing expected risks across decisions. Our approach combines experimenter-defined loss functions with observed evidence, using hierarchical models to leverage historical experiment learnings for prior information on treatment effects. Through real and simulated Amazon supply chain experiments, we demonstrate that compared to null hypothesis statistical testing, our method increases estimation efficiency via informative hierarchical priors and simplifies decision-making by systematically incorporating business preferences and costs for comprehensive, scalable decisions.
Paper Structure (14 sections, 20 equations, 6 figures, 7 tables)

This paper contains 14 sections, 20 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Decision-Making Framework
  3. Background
  4. Formulation
  5. Calculating Expected Risks
  6. Prior Construction
  7. Numerical Studies
  8. Priors with Varying Degrees of Shrinkage
  9. Posteriors with Varying Degrees of Shrinkage
  10. Estimation Efficiency on Real and Simulated Data
  11. Risks Under Different Trade-offs
  12. Defining Loss (Trade-offs) and automation
  13. Discussion
  14. Acknowledgements

Figures (6)

  • Figure 1: A hierarchical model for treatment effects. $\bm{\mu}$ and $\bm{\Gamma}$ represents the mean and the variance of the treatment effect distribution across similar experiments. $\textbf{w}_i$ represents the unobserved treatment effects of experiment $i$, and $\textbf{x}_i$ represents the results (estimated treatment effects) we observe in experiment $i$. We are interested in the distribution of $\textbf{w}_t$ given information $x_1, \cdots, x_{t-1}$.
  • Figure 2: Prior mean and standard deviation as a function of $k$. $k = 1$ produces an unbiased estimate for the covariance.
  • Figure 3: Treatment effects estimates and corresponding 95% credible intervals under different choices of $k$
  • Figure 4: Mean coverage across 20 experiments under different choices of $k$.
  • Figure 5: Decision space for E1. $x$-axis and $y$-axis represent the trade-offs between M1 and M2. For example, the point corresponding to (20, -100) on the graph represent the expected risk given trade-off vector (20, -100).
  • ...and 1 more figures