Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Measuring Sell Side Outcomes in Buy Side Marketplace Experiments using In-Experiment Bipartite Graph

Vaiva Pilkauskaitė, Jevgenij Gamper, Rasa Giniūnaitė, Agne Reklaitė

TL;DR

The paper addresses measuring seller-side outcomes from buy-side interventions in two-sided marketplaces with interference. It introduces constructing bipartite graphs from in-experiment data (e.g., item views and favorites) rather than relying on historical graphs, enabling mediation-style causal inference. It evaluates three estimators—ERL, regression-based, and CR-ERL—finding that CR-ERL offers the narrowest confidence intervals and robustness, while the choice of mediator (views vs. favorites) materially affects estimated effects. The study, grounded in a large Vinted buyer-side experiment, demonstrates practical feasibility and highlights mediation analysis as a key consideration for marketplace experimentation.

Abstract

In this study, we evaluate causal inference estimators for online controlled bipartite graph experiments in a real marketplace setting. Our novel contribution is constructing a bipartite graph using in-experiment data, rather than relying on prior knowledge or historical data, the common approach in the literature published to date. We build the bipartite graph from various interactions between buyers and sellers in the marketplace, establishing a novel research direction at the intersection of bipartite experiments and mediation analysis. This approach is crucial for modern marketplaces aiming to evaluate seller-side causal effects in buyer-side experiments, or vice versa. We demonstrate our method using historical buyer-side experiments conducted at Vinted, the largest second-hand marketplace in Europe with over 80M users.

Towards Measuring Sell Side Outcomes in Buy Side Marketplace Experiments using In-Experiment Bipartite Graph

TL;DR

The paper addresses measuring seller-side outcomes from buy-side interventions in two-sided marketplaces with interference. It introduces constructing bipartite graphs from in-experiment data (e.g., item views and favorites) rather than relying on historical graphs, enabling mediation-style causal inference. It evaluates three estimators—ERL, regression-based, and CR-ERL—finding that CR-ERL offers the narrowest confidence intervals and robustness, while the choice of mediator (views vs. favorites) materially affects estimated effects. The study, grounded in a large Vinted buyer-side experiment, demonstrates practical feasibility and highlights mediation analysis as a key consideration for marketplace experimentation.

Abstract

In this study, we evaluate causal inference estimators for online controlled bipartite graph experiments in a real marketplace setting. Our novel contribution is constructing a bipartite graph using in-experiment data, rather than relying on prior knowledge or historical data, the common approach in the literature published to date. We build the bipartite graph from various interactions between buyers and sellers in the marketplace, establishing a novel research direction at the intersection of bipartite experiments and mediation analysis. This approach is crucial for modern marketplaces aiming to evaluate seller-side causal effects in buyer-side experiments, or vice versa. We demonstrate our method using historical buyer-side experiments conducted at Vinted, the largest second-hand marketplace in Europe with over 80M users.
Paper Structure (10 sections, 7 equations, 3 figures)

This paper contains 10 sections, 7 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Related Work
  4. Constructing bipartite graph using in-experiment data
  5. Effect Estimation Methods
  6. First Approach: The Exposure-Reweighted Linear Estimator
  7. Second Approach: Regression Based Estimator
  8. Third Approach: Covariate-adjusted Exposure Reweighted Linear Estimator
  9. Experimental data and results
  10. Conclusions and Future Work

Figures (3)

  • Figure 1: Bipartite Graph adapted for Vinted Marketplace, treated buyers can interact with many sellers and their items, thus violating the SUTVA assumption. Exposure Score $H_i(Z)$ (\ref{['eq:exposureH']}) can be calculated for both Items and Sellers.
  • Figure 2: Sellers' Exposure Score distribution example for On and Off experiment. The distribution depends on how many interactions between buyers and sellers happened, as well as on the probability of buyers being from different test variants. With many sellers items receiving, for instance, one interaction from buyer, fewer receiving two, and so on - it is expected that an exposure frequency of 0 or 1 is the most common, followed by 0.5
  • Figure 3: Comparison of estimates and confidence intervals for different methods applied to two bipartite graphs with different buyer interaction events in the same experiment for two metrics (continuous and conversion). Methods include: Reg (Regression), RegB (Bootstrapped Regression), ERL+B (Bootstrapped ERL Confidence Intervals), ERL+K (Confidence Interval Estimation with Randomization Algorithm). "Pre" indicates the inclusion of a pre-experiment covariate.