Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Seller-Side Experiments under Interference Induced by Feedback Loops in Two-Sided Platforms

Zhihua Zhu, Zheng Cai, Liang Zheng, Nian Si

TL;DR

The paper develops a mathematical framework to study seller-side experiments on two-sided platforms under interference induced by feedback loops, particularly in pacing-enabled environments. It analyzes naive and counterfactual interleaving designs, showing that feedback loops can bias causal estimates and that counterfactual interleaving often underestimates the true Global Treatment Effect ($GTE$) in the presence of damping. The authors also propose a practical interference-detection method and validate insights with real-world Tencent A/B data, where observed interference caused discrepancies between estimated and actual effects. The work advances understanding of how dynamic ranking and budget controls affect experimental validity and suggests directions for robust design under feedback-loop dynamics with significant practical impact for platforms balancing exploration, exploitation, and resource constraints.

Abstract

Two-sided platforms are central to modern commerce and content sharing and often utilize A/B testing for developing new features. While user-side experiments are common, seller-side experiments become crucial for specific interventions and metrics. This paper investigates the effects of interference caused by feedback loops on seller-side experiments in two-sided platforms, with a particular focus on the counterfactual interleaving design, proposed in \citet{ha2020counterfactual,nandy2021b}. These feedback loops, often generated by pacing algorithms, cause outcomes from earlier sessions to influence subsequent ones. This paper contributes by creating a mathematical framework to analyze this interference, theoretically estimating its impact, and conducting empirical evaluations of the counterfactual interleaving design in real-world scenarios. Our research shows that feedback loops can result in misleading conclusions about the treatment effects.

Seller-Side Experiments under Interference Induced by Feedback Loops in Two-Sided Platforms

TL;DR

The paper develops a mathematical framework to study seller-side experiments on two-sided platforms under interference induced by feedback loops, particularly in pacing-enabled environments. It analyzes naive and counterfactual interleaving designs, showing that feedback loops can bias causal estimates and that counterfactual interleaving often underestimates the true Global Treatment Effect () in the presence of damping. The authors also propose a practical interference-detection method and validate insights with real-world Tencent A/B data, where observed interference caused discrepancies between estimated and actual effects. The work advances understanding of how dynamic ranking and budget controls affect experimental validity and suggests directions for robust design under feedback-loop dynamics with significant practical impact for platforms balancing exploration, exploitation, and resource constraints.

Abstract

Two-sided platforms are central to modern commerce and content sharing and often utilize A/B testing for developing new features. While user-side experiments are common, seller-side experiments become crucial for specific interventions and metrics. This paper investigates the effects of interference caused by feedback loops on seller-side experiments in two-sided platforms, with a particular focus on the counterfactual interleaving design, proposed in \citet{ha2020counterfactual,nandy2021b}. These feedback loops, often generated by pacing algorithms, cause outcomes from earlier sessions to influence subsequent ones. This paper contributes by creating a mathematical framework to analyze this interference, theoretically estimating its impact, and conducting empirical evaluations of the counterfactual interleaving design in real-world scenarios. Our research shows that feedback loops can result in misleading conclusions about the treatment effects.
Paper Structure (16 sections, 5 theorems, 66 equations, 4 figures, 3 tables)

This paper contains 16 sections, 5 theorems, 66 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Literature
  3. Interference in Experiments
  4. Feedback Loops in Recommendation Systems
  5. Seller-Side Experiments
  6. A Model for seller-side experiments under Interference Induced by Feedback Loops
  7. Recommendation System Pipeline
  8. Seller-Side Experiments without Feedback Loops
  9. Naive seller-side experiments
  10. Counterfactual interleaving design.
  11. Seller-Side Experiments with Feedback Loops
  12. Naive seller-side experiments
  13. Counterfactual interleaving design.
  14. Empirical Results based on Real-World A/B tests
  15. Conclusion
  16. ...and 1 more sections

Key Result

Proposition 1

Suppose that $\hat{e}_i^C(t)=\hat{e}_i^T(t)$ for $i=1,2,\ldots, T$ and $t\in[0,H]$. Then, we have the GTE estimator (GTE_hat) is unbiased of the true GTE (GTE), i.e., $\mathbb{E}[\widehat{GTE}]=\mathrm{GTE}.$

Figures (4)

  • Figure 1: A standard pipeline in platforms with a pacing algorithm.
  • Figure 2: Counterfactual interleave design.
  • Figure 3: Dependence of different objects in the feedback loops.
  • Figure 4: Average $\lambda^T$ values, $\bar{\lambda}^T$s, for the control ads, treatment ads, and other ads over time in a day.

Theorems & Definitions (10)

  • Proposition 1
  • Remark 1
  • Proposition 2
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • proof : Proof of Proposition \ref{['prop:NEwithoutcase1']}
  • proof : Proof of Proposition \ref{['prop:CEwithoutunbiased']}
  • proof : Proof of Theorem \ref{['thm:NE:with']}
  • proof : Proof of Theorem \ref{['thm:CE:with2']}