A Two-Part Machine Learning Approach to Characterizing Network Interference in A/B Testing

TL;DR

This work tackles the bias and variance challenges caused by network interference in A/B testing by proposing a two-part machine learning framework that automates exposure mapping. It combines causal network motifs (g) to form motif representations and a clustering-based mapper (h) to define exposure regions, enabling robust estimation of average potential outcomes and the global average treatment effect with Horvitz–Thompson or Hájek estimators. The method is validated through synthetic Watts-Strogatz and Slashdot networks and a large-scale Instagram A/B test, showing reduced bias and improved inference relative to traditional design-based and exposure-mapping approaches. Practically, the approach offers a scalable, interpretable, and automated tool for practitioners to diagnose interference patterns, inform experiment design, and improve decision-making in marketing and product optimization.

Abstract

The reliability of controlled experiments, commonly referred to as "A/B tests," is often compromised by network interference, where the outcomes of individual units are influenced by interactions with others. Significant challenges in this domain include the lack of accounting for complex social network structures and the difficulty in suitably characterizing network interference. To address these challenges, we propose a machine learning-based method. We introduce "causal network motifs" and utilize transparent machine learning models to characterize network interference patterns underlying an A/B test on networks. Our method's performance has been demonstrated through simulations on both a synthetic experiment and a large-scale test on Instagram. Our experiments show that our approach outperforms conventional methods such as design-based cluster randomization and conventional analysis-based neighborhood exposure mapping. Our approach provides a comprehensive and automated solution to address network interference for A/B testing practitioners. This aids in informing strategic business decisions in areas such as marketing effectiveness and product customization.

Figures (16)

• Figure 1: Diagram of our two-part approach compared to the conventional exposure mapping framework
• Figure 2: Illustration of causal network motifs. (a) Examples of causal network motifs. Solid nodes indicate treatment, hollow nodes indicate control, and shaded nodes indicate that they could be in treatment or control. The star node for each network is the ego. The first pattern in each row represents conventional network motifs without assignment conditions, followed by corresponding causal network motifs. Our causal network motif representation is constructed by dividing the count of a causal network motif by the count of the corresponding network motif. The labels below each network motif indicate the naming: for example, an open triad where one neighbor is treated is named 3o-1 (the second causal network motif of the third row). (b) Construction of causal network motif representation. This illustration represents an ego network with treatment assignments and its corresponding causal network motif representation, considering $1$-hop network interference. The first dimension corresponds to the random assignment received by unit $i$ (denoted as $Z_{i1}$). Each subsequent dimension represents a causal network motif, calculated as the number of a specific causal network motif divided by the total number of corresponding network motifs. Random noise ($U_{im}$) is introduced to adjust the values so that each dimension maintains full support within the range $[0, 1]$.
• Figure 3: Results for the WS synthetic experiment with the baseline potential outcomes: (Upper) Hájek and (Lower) HT estimators. Blue and pink error bars represent Bernoulli and graph cluster randomization, respectively. From left to right, the panels illustrate the estimators for the counterfactual world where every unit is treated ($\mu(\mathbf{1})$), non-treated ($\mu(\mathbf{0})$), and the global average treatment effects ($\tau$). The $x$-label $K/N$ represents the fraction of the number of nodes used in the nearest neighbor exposure condition relative to the entire population. Error bars depict standard errors, wherein the dotted error bars indicate estimates that do not meet the positivity requirement. The dashed gray lines represent the ground truth. The label "avg" denotes the approach of calculating average outcomes for treatment and/or control groups.
• Figure 4: Results for the WS synthetic experiment illustrating the base potential outcomes model across different sets of causal network motifs and distance metrics (Hájek estimators). Each panel depicts estimators for the global average treatment effects ($\tau$) for different sets of causal network motifs, including the benchmark ($q$-frac). Various curves represent different distance metrics. Other visual elements follow conventions in Fig. \ref{['fig:main']}.
• Figure 5: Results for the WS synthetic experiment illustrating parameterized potential outcomes under different values for $\lambda$. Each panel depicts estimators for the global average treatment effects ($\tau$) for different sets of causal network motifs, including the benchmark ($q$-frac). Various curves within the panels represent different choices of $\lambda$. Other visual elements follow conventions in Fig. \ref{['fig:main']}.

Theorems & Definitions (12)

• Definition 1: $n$-Hop Ego Network
• Definition 2: Exposure Mapping
• Definition 3: Correctly Specified Exposure Mapping
• Definition 4: Positivity Requirement
• Lemma 1
• Definition 5: Representation Invariant to $\bm{z}$
• Remark 1
• Proposition 1
• Definition 6: Properly Specified Distance Metric
• Proposition 2