Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Randomization Inference in Two-Sided Market Experiments

Jizhou Liu, Azeem M. Shaikh, Panos Toulis

Abstract

Randomized experiments are increasingly employed in two-sided markets, such as buyer--seller platforms, to evaluate the effects of marketplace interventions. These experiments must reflect the underlying two-sided market structure in their design and can therefore be challenging to analyze. In this paper, we develop a randomization inference framework for outcomes from two-sided experiments, with a focus on testing and inference for two-sided spillover effects. Our approach is finite-sample valid under sharp null hypotheses. Regarding weak null hypotheses, we find that the commonly used Neyman-style studentization does not universally ensure asymptotic validity, and we document how it depends on the specific formulation of the null. We then propose a two-way variance estimator for studentization that restores asymptotic validity. We further propose methods to improve testing power by exploiting the two-sided structure of the problem, which we validate empirically. We demonstrate our methods through a series of simulation studies and an applied example from a network experiment in micro-lending.

Randomization Inference in Two-Sided Market Experiments

Abstract

Randomized experiments are increasingly employed in two-sided markets, such as buyer--seller platforms, to evaluate the effects of marketplace interventions. These experiments must reflect the underlying two-sided market structure in their design and can therefore be challenging to analyze. In this paper, we develop a randomization inference framework for outcomes from two-sided experiments, with a focus on testing and inference for two-sided spillover effects. Our approach is finite-sample valid under sharp null hypotheses. Regarding weak null hypotheses, we find that the commonly used Neyman-style studentization does not universally ensure asymptotic validity, and we document how it depends on the specific formulation of the null. We then propose a two-way variance estimator for studentization that restores asymptotic validity. We further propose methods to improve testing power by exploiting the two-sided structure of the problem, which we validate empirically. We demonstrate our methods through a series of simulation studies and an applied example from a network experiment in micro-lending.

Paper Structure

This paper contains 39 sections, 8 theorems, 201 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Setup and Notation
  3. Main Method
  4. Overview of Conditional Randomization Tests
  5. Testing spillover effects
  6. Testing Total effects
  7. Testing Procedure with $k$-Block Conditioning Event
  8. Selecting $k$ Based on Power Considerations
  9. Randomization Tests for Weak Null Hypotheses
  10. Standard Neyman-style studentization
  11. Two-way studentization
  12. Simulations
  13. Sharp null hypotheses
  14. Weak null hypotheses for buyer spillover effects
  15. Data Application
  16. ...and 24 more sections

Key Result

Theorem 3.1

Consider an independent two-sided randomized design (Definition def:multiple-randomization) where Assumption ass:local-interference holds. Let $\mathcal{C}$ be a buyer spillover conditioning event, as defined in Equation eq:cond_buyer.

Figures (5)

  • Figure 1: Graphical illustration of the conditioning event for Procedure \ref{['procedure:spillover']}. The shaded area illustrates that the procedure conditions on control sellers (columns), and then randomizes the treatments on buyers (rows).
  • Figure 2: Graphical illustration of the conditioning event for Procedure \ref{['procedure:total']}. The shaded area illustrates that the procedure permutes the treatment assignments across diagonal blocks.
  • Figure 3: Graphical illustration of the lower bound on power for varying $k$ with $n = 400$, $A = 1$, and $a = \tau = 0.01$.
  • Figure 4: Limit distribution of Neyman-style studentized statistic under $H_0^{wb,1}: \tau = 0$
  • Figure 5: All available variables from the data

Theorems & Definitions (23)

  • Definition 2.1: Independent Two-Sided Randomized Design
  • Example 2.1
  • Example 2.2: Two-Sided Design under Complete Randomization
  • Example 2.3
  • Remark 3.1
  • Remark 3.2: Randomization tests when design symmetry fails
  • Remark 3.3: Seller spillover hypothesis
  • Theorem 3.1
  • Definition 3.1: $k$-Block Conditioning Event
  • Theorem 3.2
  • ...and 13 more