Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Post-Experiment Decisions: The Dual Adjustments for Rollout and Downstream Optimizations

Guoxing He, Dan Yang, Wei Zhang

Abstract

Firms increasingly use randomized experiments to decide whether to scale up an intervention and, if so, how to re-optimize related operational choices such as inventory, capacity, or pricing. In many settings, experiments are performed on small samples, so the estimated effect of the intervention is uncertain. A common practice is to plug a 'significant' estimate of the effect into both (i) the rollout rule and (ii) the downstream optimization. However, this can lead to avoidable losses because the costs of over- versus under-estimating the effect are often asymmetric. The technically ideal approach is to obtain a data-dependent decision rule that minimizes the Bayes risk, but this lacks transparency and requires more computations. We propose Predict-Adjust-Then-Rollout-Optimize (PATRO), a plug-in approach that keeps the standard estimate, but makes data-independent adjustments, respectively, for the two types of decision. We show that the two adjustments can be substitutes or complements and provide an alternating-iteration method to compute the pair. PATRO performs both in theory and numerically close or equivalent to the Bayes-optimal benchmark, making it a simple, effective way to convert noisy experimental results into better rollout and operational decisions.

Post-Experiment Decisions: The Dual Adjustments for Rollout and Downstream Optimizations

Abstract

Firms increasingly use randomized experiments to decide whether to scale up an intervention and, if so, how to re-optimize related operational choices such as inventory, capacity, or pricing. In many settings, experiments are performed on small samples, so the estimated effect of the intervention is uncertain. A common practice is to plug a 'significant' estimate of the effect into both (i) the rollout rule and (ii) the downstream optimization. However, this can lead to avoidable losses because the costs of over- versus under-estimating the effect are often asymmetric. The technically ideal approach is to obtain a data-dependent decision rule that minimizes the Bayes risk, but this lacks transparency and requires more computations. We propose Predict-Adjust-Then-Rollout-Optimize (PATRO), a plug-in approach that keeps the standard estimate, but makes data-independent adjustments, respectively, for the two types of decision. We show that the two adjustments can be substitutes or complements and provide an alternating-iteration method to compute the pair. PATRO performs both in theory and numerically close or equivalent to the Bayes-optimal benchmark, making it a simple, effective way to convert noisy experimental results into better rollout and operational decisions.
Paper Structure (35 sections, 12 theorems, 56 equations, 4 figures, 1 table, 1 algorithm)

This paper contains 35 sections, 12 theorems, 56 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Predict-then-Optimize Framework and Post-Estimation Adjustment
  4. Causal Decision-Making, Policy Learning, and Experimentation
  5. The Model
  6. Effect Estimation in the Bayesian Framework
  7. Group means and sampling distribution.
  8. Likelihood and Prior.
  9. Posterior.
  10. The Decision-Making Framework
  11. Regret.
  12. Decision-Making Adjustments
  13. Adjustment for the Rollout Decision
  14. Quantile-Based Adjustment Rule
  15. The Optimal Adjustment
  16. ...and 20 more sections

Key Result

Lemma 1

The PTO estimator $\tilde{m}$ satisfies the following properties:

Figures (4)

  • Figure 1: The PATRO Workflow
  • Figure 2: Inventory Management under Demand Uncertainty
  • Figure 3: Service Operations with Capacity Planning
  • Figure 4: Pricing with Log-Linear Demand

Theorems & Definitions (13)

  • Lemma 1
  • Proposition 1: Optimal $\delta_n^r$
  • Proposition 2: Optimal $\delta^o_n$
  • Corollary 1: Necessity of Adjustment for the Operational Decisions
  • Corollary 2: Direction of Adjustment for the Operational Decisions
  • Proposition 3: Dual Adjustments
  • Corollary 3
  • Corollary 4
  • Definition 1: Substitution and Complementarity of Adjustments
  • Proposition 4: Conditions for Complementarity
  • ...and 3 more