Interference Among First-Price Pacing Equilibria: A Bias and Variance Analysis

TL;DR

The paper tackles bias from interference in budget-constrained A/B tests in online ad markets by introducing a parallel, submarket-based budget-controlled design. It formalizes market interference within the first-price pacing equilibrium framework, and develops a debiased surrogate for pacing multipliers together with a plug-in estimator that achieves asymptotic normality. The authors show how to perform statistical inference under supply contamination and demonstrate through semi-synthetic experiments that the debiasing procedure reduces first-order bias and yields credible confidence intervals. The approach increases A/B testing throughput in large marketplaces while controlling interference, with practical guidelines for submarket clustering and inference. The results contribute both a practical experimental design and a theoretical toolkit for FPPE-based inference under interference.

Abstract

Online A/B testing is widely used in the internet industry to inform decisions on new feature roll-outs. For online marketplaces (such as advertising markets), standard approaches to A/B testing may lead to biased results when buyers operate under a budget constraint, as budget consumption in one arm of the experiment impacts performance of the other arm. To counteract this interference, one can use a budget-split design where the budget constraint operates on a per-arm basis and each arm receives an equal fraction of the budget, leading to ``budget-controlled A/B testing.'' Despite clear advantages of budget-controlled A/B testing, performance degrades when budget are split too small, limiting the overall throughput of such systems. In this paper, we propose a parallel budget-controlled A/B testing design where we use market segmentation to identify submarkets in the larger market, and we run parallel experiments on each submarket. Our contributions are as follows: First, we introduce and demonstrate the effectiveness of the parallel budget-controlled A/B test design with submarkets in a large online marketplace environment. Second, we formally define market interference in first-price auction markets using the first price pacing equilibrium (FPPE) framework. Third, we propose a debiased surrogate that eliminates the first-order bias of FPPE, drawing upon the principles of sensitivity analysis in mathematical programs. Fourth, we derive a plug-in estimator for the surrogate and establish its asymptotic normality. Fifth, we provide an estimation procedure for submarket parallel budget-controlled A/B tests. Finally, we present numerical examples on semi-synthetic data, confirming that the debiasing technique achieves the desired coverage properties.

Figures (6)

• Figure 1: Parallel vs. standard budget-controlled A/B test, daily treatment effect. We denote neutral treatment effects with a value of $1.0$. Red crosses indicate instances of sign inconsistencies. Left are all datapoints, on the right, datapoints that fail a guardrail metric are removed.
• Figure 2: Left: Finite FPPE (left) and limit FPPE (right). In a finite FPPE, there are a finite number of items; in a limit FPPE, the item set is a continuum. Right: The interference model --- Left ($\widehat{{\mathscr{M}}}_\alpha$): the observed market where interference is present among submarkets. Middle (${\mathscr{M}}_\alpha$): the limit market with interference from bad item set. Right (${\mathscr{M}}_0$): the limit market with perfectly separated submarkets. We use data from the left panel to make inferences about the market in the right panel.
• Figure 3: Normalized bias (in percent of true value) as a function of $\alpha$ in semi-synthetic experiments. $\widetilde{\boldsymbol \beta}^*$ and $\widetilde{{ {{REV}}}}^*$ are the debiased surrogates for pacing multiplier and revenue in the limit market with interference ${\mathscr{M}}_\alpha$.
• Figure 4: Revenue confidence intervals as a function of the number of items in semi-synthetic experiments. The analytic CI comes from \ref{['eq: revenue_ci']}. The true value is the debiased surrogates for revenue in the limit market with interference ${\mathscr{M}}_\alpha$.
• Figure 5: Normalized bias (in percent of true value) as a function of $\alpha$ in fully synthetic experiments. $\widetilde{\boldsymbol \beta}^*$ and $\widetilde{{ {{REV}}}}^*$ are the debiased surrogates for pacing multiplier and revenue in the limit market with interference ${\mathscr{M}}_\alpha$.

Theorems & Definitions (12)

• Definition 1: Limit FPPE, gao2022infiniteconitzer2022pacing
• Definition 2: SMO
• Definition 3: SCS
• Definition 4: Finite FPPE, informal
• Theorem 1: Analysis of Bias
• Theorem 2: Consistency
• Theorem 3
• Lemma 1: Theorem 1 from shapiro1990differential
• Lemma 2: Theorem 2.2 from stewart1977perturbation
• proof