Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reducing Marketplace Interference Bias Via Shadow Prices

Ido Bright, Arthur Delarue, Ilan Lobel

TL;DR

This work proposes a technique for platforms to run standard RCTs and still obtain meaningful estimates despite the presence of marketplace interference, and proves that the estimator is less biased than the RCT estimator.

Abstract

Marketplace companies rely heavily on experimentation when making changes to the design or operation of their platforms. The workhorse of experimentation is the randomized controlled trial (RCT), or A/B test, in which users are randomly assigned to treatment or control groups. However, marketplace interference causes the Stable Unit Treatment Value Assumption (SUTVA) to be violated, leading to bias in the standard RCT metric. In this work, we propose techniques for platforms to run standard RCTs and still obtain meaningful estimates despite the presence of marketplace interference. We specifically consider a generalized matching setting, in which the platform explicitly matches supply with demand via a linear programming algorithm. Our first proposal is for the platform to estimate the value of global treatment and global control via optimization. We prove that this approach is unbiased in the fluid limit. Our second proposal is to compare the average shadow price of the treatment and control groups rather than the total value accrued by each group. We prove that this technique corresponds to the correct first-order approximation (in a Taylor series sense) of the value function of interest even in a finite-size system. We then use this result to prove that, under reasonable assumptions, our estimator is less biased than the RCT estimator. At the heart of our result is the idea that it is relatively easy to model interference in matching-driven marketplaces since, in such markets, the platform mediates the spillover.

Reducing Marketplace Interference Bias Via Shadow Prices

TL;DR

This work proposes a technique for platforms to run standard RCTs and still obtain meaningful estimates despite the presence of marketplace interference, and proves that the estimator is less biased than the RCT estimator.

Abstract

Marketplace companies rely heavily on experimentation when making changes to the design or operation of their platforms. The workhorse of experimentation is the randomized controlled trial (RCT), or A/B test, in which users are randomly assigned to treatment or control groups. However, marketplace interference causes the Stable Unit Treatment Value Assumption (SUTVA) to be violated, leading to bias in the standard RCT metric. In this work, we propose techniques for platforms to run standard RCTs and still obtain meaningful estimates despite the presence of marketplace interference. We specifically consider a generalized matching setting, in which the platform explicitly matches supply with demand via a linear programming algorithm. Our first proposal is for the platform to estimate the value of global treatment and global control via optimization. We prove that this approach is unbiased in the fluid limit. Our second proposal is to compare the average shadow price of the treatment and control groups rather than the total value accrued by each group. We prove that this technique corresponds to the correct first-order approximation (in a Taylor series sense) of the value function of interest even in a finite-size system. We then use this result to prove that, under reasonable assumptions, our estimator is less biased than the RCT estimator. At the heart of our result is the idea that it is relatively easy to model interference in matching-driven marketplaces since, in such markets, the platform mediates the spillover.
Paper Structure (48 sections, 26 theorems, 152 equations, 13 figures)

This paper contains 48 sections, 26 theorems, 152 equations, 13 figures.

Table of Contents

  1. Introduction
  2. Related Literature
  3. The Model
  4. The Platform
  5. Supply and demand types.
  6. Special case: bipartite matching.
  7. General case: matching via network flows.
  8. The Experimental Setting
  9. The Experimental Data
  10. Bipartite matching.
  11. General matching.
  12. Interchangeability of demand and control units.
  13. The Standard Estimator and Interference Bias
  14. The RCT Estimator
  15. The Fluid Limit
  16. ...and 33 more sections

Key Result

Theorem 1

For a given demand rate $\bm \lambda$, the fluid limit of the average total value is given by Further, let $X^{\tau}_{u,w}$ and $(A^{\tau}_i, B^{\tau}_j, M^{\tau}_w,\Xi^\tau_{u,w})$correspond to the optimal primal and dual solutions of the matching LP $\Phi_\tau\left(\frac{1}{\tau}\mathbf{D}^{\tau,\bm \lambda},\frac{1}{\tau}\mathbf{S}^{\tau}\right)$. Then $X^{\tau}_{u,w}$ converges almost su

Figures (13)

  • Figure 1: Diagram of our matching setting. The general setting simply replaces the direct edges between demand and supply with an arbitrary directed flow network.
  • Figure 2: Illustrative bipartite matching example with a single demand type ($n_d=1$) and six supply types ($n_s=6$). The demand arrival rate is $\lambda_1=1.5$ under global control, and $\lambda_1+\beta=5.5$ under global treatment. The supply arrival rate is 1 for each type. The matching values of the unique demand type decrease geometrically for each supply type, creating contention for the highest-value supply units of type 1. As a result, the matching value function (solid line) is concave in the demand arrival rate; it also has seven linear pieces, with slopes $v_{1,1}, \ldots, v_{1,6}, 0$.
  • Figure 3: Bias of the RCT estimator in the example from Figure \ref{['fig:example-setup']}. The estimator implicitly constructs a linear approximation (dashed line) of the value function based on the observed value at the experiment state, where the demand arrival rate is $\lambda_1 + \rho\beta$. Bias occurs because this linear approximation fails to account for concavity.
  • Figure 4: Example of a situation where the Two-LP estimator fails. We set up a capacitated matching instance where edges between the $i$-th demand and $i$-th supply type have value and capacity one, while all other edges have value zero. Treatment doubles the demand arrival rates, leading to a global treatment effect of 2. However, the Two-LP estimator constructed from a 50-50 experiment doubles the units instead of the rates, underestimating both control and treatment values (and thus, the global treatment effect) by a factor of two.
  • Figure 5: Linear approximation corresponding to the SP estimator.
  • ...and 8 more figures

Theorems & Definitions (56)

  • Definition 1
  • Definition 2
  • Theorem 1
  • Corollary 1
  • Definition 3
  • Proposition 1
  • Proposition 2
  • Theorem 2
  • Definition 4: Two-LP estimator
  • Proposition 3
  • ...and 46 more