Price Experimentation and Interference

TL;DR

This paper analyzes biases that arise when platforms run A/B tests on a continuous variable, notably prices, to infer global treatment effects on metrics like demand and profits. Using a structural model and differential calculus, it shows that interference between market participants can produce a nontrivial bias that may flip the sign of profit effects, potentially leading firms to move prices in the wrong direction. It develops a simple debiasing approach requiring a 50/50 allocation between treatment and control, and applies the framework to a two-sided market with mean-field dynamics, deriving conditions under which sign flips occur and quantifying the resulting losses relative to the optimal pricing. An empirical Airbnb calibration demonstrates that wrong-sign estimators are plausible in practice, underscoring the practical relevance and need for caution in price experimentation. The work thus provides theoretical insights, practical debiasing tools, and empirical validation for price experimentation in platforms and marketplaces.

Abstract

In this paper, we examine the biases that arise when firms run A/B tests on continuous parameters to estimate global treatment effects on performance metrics of interest; we particularly focus on price experiments to measure the price impact on quantity demanded, and on profit. In canonical A/B experimental estimators, biases emerge due to interference between market participants. We employ structural modeling and differential calculus to derive intuitive characterizations of these biases. We then specialize our general model to the standard revenue-management pricing problem. This setting highlights a fundamental risk innate to A/B pricing experiments: that the canonical estimator for the expected change in profits, counterintuitively, can have the wrong sign in expectation. In other words, following the guidance of canonical estimators may lead firms to move prices (or fees) in the wrong direction, inadvertently decreasing profits. We introduce a novel debiasing technique for these canonical experiments, requiring only that firms equally split units between treatment and control. We apply these results to a two-sided market model, and demonstrate how the "change of sign" regime depends on market factors such as the supply/demand imbalance, and the price markup. We conclude by calibrating our revenue-management pricing model to published empirical estimates from Airbnb marketplaces, demonstrating that estimators with the wrong sign are not a knife-edge issue, and that they may be prevalent enough to be of concern to practitioners.

Figures (9)

• Figure 1: Illustration of $Bias^{LR}_D/\lambda$ and $Bias^{LR}_\pi/\lambda$ as a function of price ($p$) and market balance ($\lambda$). Instance is given by $v(p)=e^{V-p}$, $\epsilon = 1$, $c = 1$, $V=5$, and $\rho = 1$.
• Figure 2: Illustration of $Bias^{CR}_D/\lambda$ and $Bias^{LR}_\pi/\lambda$ as a function of price ($p$) and market balance ($\lambda$). Instance is given by $v(p)=e^{V-p}$, $\epsilon = 1$, $c = 1$, $V=5$, and $\rho = 1$.
• Figure 3: Illustration of the change of sign region (purple) in an LR-experiment (left) and CR-experiment (right), respectively, with $v(p)=e^{V-p}$, $V=5$, $c=0.5$, $\rho=1$, and $\epsilon=1$. The vertical axis is the (log of) the arrival rate, or market balance, parameter $\lambda$; the horizontal axis is the (baseline) control price $p$, at which the experiment is conducted. The region where condition (a) is not satisfied ($GTE_{\pi}< 0$), so there is no change of sign, is in red. The region where condition (b) is not satisfied ($Bias_{\pi}^{LR} < GTE_\pi$ or $Bias_{\pi}^{CR} < GTE_\pi$, respectively), so there is no change of sign, is in blue.
• Figure 4: Illustration of the gap from optimal profits that canonical LR, and CR A/B experimentation yields, with $v(p) = e^{V-p}$, $V=5$, $c=1$, $\rho=1$, and $\epsilon=1$.
• Figure 5: Change of sign region for price and cross-price elasticity in the Airbnb Boston Economy Segment. Fradkin2022 estimates $\hat{c} = \$81.4$, and $\hat{e}_p(p): -2.33$.

Theorems & Definitions (8)

• Proposition 1
• Proposition 2
• Proposition 3
• Proposition 4
• Theorem 1
• Theorem 2
• Proposition 5
• Proposition 6