Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Validating Causal Message Passing Against Network-Aware Methods on Real Experiments

Albert Tan, Sadegh Shirani, James Nordlund, Mohsen Bayati

TL;DR

The paper tackles interference in randomized experiments by validating causal message passing (CMP), a network-free method that exploits temporal outcome dynamics, against a network-aware bipartite benchmark on two real field experiments. CMP achieves estimates that align in direction and statistical significance with the network-aware approach for the primary decision metric, despite not observing the underlying interaction network. The study shows that temporal variation provides sufficient information to recover the bias direction of spillovers and, in at least one case, even enhances statistical power relative to network-based methods. The findings suggest practitioners can leverage CMP when network data are costly, proprietary, or unreliable, enabling robust estimation of spillover effects from temporal dynamics alone.

Abstract

Estimating total treatment effects in the presence of network interference typically requires knowledge of the underlying interaction structure. However, in many practical settings, network data is either unavailable, incomplete, or measured with substantial error. We demonstrate that causal message passing, a methodology that leverages temporal structure in outcome data rather than network topology, can recover total treatment effects comparable to network-aware approaches. We apply causal message passing to two large-scale field experiments where a recently developed bipartite graph methodology, which requires network knowledge, serves as a benchmark. Despite having no access to the interaction network, causal message passing produces effect estimates that match the network-aware approach in direction across all metrics and in statistical significance for the primary decision metric. Our findings validate the premise of causal message passing: that temporal variation in outcomes can serve as an effective substitute for network observation when estimating spillover effects. This has important practical implications: practitioners facing settings where network data is costly to collect, proprietary, or unreliable can instead exploit the temporal dynamics of their experimental data.

Validating Causal Message Passing Against Network-Aware Methods on Real Experiments

TL;DR

The paper tackles interference in randomized experiments by validating causal message passing (CMP), a network-free method that exploits temporal outcome dynamics, against a network-aware bipartite benchmark on two real field experiments. CMP achieves estimates that align in direction and statistical significance with the network-aware approach for the primary decision metric, despite not observing the underlying interaction network. The study shows that temporal variation provides sufficient information to recover the bias direction of spillovers and, in at least one case, even enhances statistical power relative to network-based methods. The findings suggest practitioners can leverage CMP when network data are costly, proprietary, or unreliable, enabling robust estimation of spillover effects from temporal dynamics alone.

Abstract

Estimating total treatment effects in the presence of network interference typically requires knowledge of the underlying interaction structure. However, in many practical settings, network data is either unavailable, incomplete, or measured with substantial error. We demonstrate that causal message passing, a methodology that leverages temporal structure in outcome data rather than network topology, can recover total treatment effects comparable to network-aware approaches. We apply causal message passing to two large-scale field experiments where a recently developed bipartite graph methodology, which requires network knowledge, serves as a benchmark. Despite having no access to the interaction network, causal message passing produces effect estimates that match the network-aware approach in direction across all metrics and in statistical significance for the primary decision metric. Our findings validate the premise of causal message passing: that temporal variation in outcomes can serve as an effective substitute for network observation when estimating spillover effects. This has important practical implications: practitioners facing settings where network data is costly to collect, proprietary, or unreliable can instead exploit the temporal dynamics of their experimental data.
Paper Structure (18 sections, 7 equations, 2 figures, 2 tables)

This paper contains 18 sections, 7 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Methodology Overview
  5. Estimand: Total Treatment Effect
  6. Network-Aware Benchmark: Bipartite Graph Methodology
  7. Causal Message Passing: Temporal Dynamics Without Network Data
  8. Comparison of Approaches
  9. Experimental Setup
  10. Experimental Context
  11. Experiments A and B
  12. Metrics
  13. Implementation
  14. Results
  15. CMP Recovers Network-Aware Estimates
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Overview of our empirical validation framework. We apply three estimation methods to two real field experiments (Experiment A: $n \approx 7{,}000$; Experiment B: $n \approx 4{,}000$) conducted in a bipartite setting where treatment units interact with connected units, creating network interference. The Basic Method ignores interference and produces biased estimates. The Network-Aware approach leverages observed network data to correct for spillovers. Causal Message Passing (CMP) uses only temporal outcome dynamics, without any network information, yet produces estimates that align with Network-Aware in both direction and significance on the primary decision metric.
  • Figure 2: Bipartite structure in experimental settings. Treatment units (left) are partitioned into eligible units, which can be assigned to treatment, and ineligible units, which remain in control throughout the experiment. Connected units (right) interact with treatment units through edges representing service relationships. Outcomes are aggregated at the treatment unit level by summing edge-level outcomes. Solid edges connect eligible treatment units to connected units; dashed edges connect ineligible treatment units. In this paper, our analysis focuses on eligible treatment units, indexed $i = 1, \ldots, N$.