Treatment Effect Estimation Amidst Dynamic Network Interference in Online Gaming Experiments

TL;DR

Dynamic, ephemeral networks in online gaming violate SUTVA, motivating a causal framework for treatment effects under interference. The paper introduces a post-hoc exposure mapping using $M_i$ (the number of treated games) and an IPW-based estimator that relies on unconfoundedness given covariates to identify and estimate $\tau(m)$ and the overall $\tau$. It validates the approach via simulations across interference regimes and a real Tencent mobile-gaming dataset, showing reduced bias when incorporating both treatment and control-mixed groups. The framework generalizes to other digital platforms with evolving networks, enabling robust causal inference in settings where network structure is unknown or transient.

Abstract

The evolving landscape of online multiplayer gaming presents unique challenges in assessing the causal impacts of game features. Traditional A/B testing methodologies fall short due to complex player interactions, leading to violations of fundamental assumptions like the Stable Unit Treatment Value Assumption (SUTVA). Unlike traditional social networks with stable and long-term connections, networks in online games are often dynamic and short-lived. Players are temporarily teamed up for the duration of a game, forming transient networks that dissolve once the game ends. This fleeting nature of interactions presents a new challenge compared with running experiments in a stable social network. This study introduces a novel framework for treatment effect estimation in online gaming environments, considering the dynamic and ephemeral network interference that occurs among players. We propose an innovative estimator tailored for scenarios where a completely randomized experimental design is implemented without explicit knowledge of network structures. Notably, our method facilitates post-hoc interference adjustment on experimental data, significantly reducing the complexities and costs associated with intricate experimental designs and randomization strategies. The proposed framework stands out for its ability to accommodate varying levels of interference, thereby yielding more accurate and robust estimations. Through comprehensive simulations set against a variety of interference scenarios, along with empirical validation using real-world data from a mobile gaming environment, we demonstrate the efficacy of our approach. This study represents a pioneering effort in exploring causal inference in user-randomized experiments impacted by dynamic network effects.

Figures (5)

• Figure 1: An illustration of the randomized experiment process with network interference during online games.
• Figure 2: Visualization of the sample size proportions versus different treatment levels $M$ for the treatment (in red lines), control-mixed (in blue lines) and control-control (in yellow dots) groups with 50 simulated data sets under each simulation setting. It shows the different interference structures under different treatment player matching probabilities and number of game generations.
• Figure 3: Comparisons of the Average Treatment Effect $\tau(m)$ estimations under each treatment level $M$ for the four estimators, with the corresponding mean and 95% interval of the estimated effects under 100 simulated data sets in each simulation setting. The black line is the ground truth of $\hat{\tau}(m)$.
• Figure 4: The side-by-side boxplots of the target metric (TM) for the control-control, control-mixed and treatment groups in the pre-experiment and experiment data sets.
• Figure 5: The Average Treatment Effect $\tau(m)$ estimations under each treatment level $m$ for the four estimators based on the real online gaming data set.