Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy design in experiments with unknown interference

Davide Viviano, Jess Rudder

TL;DR

This paper develops experimental designs for policy design in settings with unknown spillovers confined to a small number of large clusters. It introduces a single-wave approach that leverages local perturbations across two clusters to identify the marginal policy effect (MPE) and to test welfare-optimality, while also enabling estimates of direct, spillover, and welfare effects. It further proposes a multi-wave adaptive design that iteratively learns welfare-maximizing treatment rules with formal guarantees on out-of-sample and in-sample regret, leveraging sequential gradient updates and sequential cross-fitting to handle unobserved interference. The authors provide rigorous theoretical results, including consistency, inference, and fast convergence rates, and implement the methodology in a large-scale field experiment in rural Pakistan (weather-forecast diffusion) with over 250,000 farmers, illustrating substantial welfare gains and practical cost savings. The work contributes to the literature on designing policies under network spillovers and partial interference, offering tractable tools for policy designers when network data are costly or unavailable.

Abstract

This paper studies experimental designs for estimation and inference on policies with spillover effects. Units are organized into a finite number of large clusters and interact in unknown ways within each cluster. First, we introduce a single-wave experiment that, by varying the randomization across cluster pairs, estimates the marginal effect of a change in treatment probabilities, taking spillover effects into account. Using the marginal effect, we propose a test for policy optimality. Second, we design a multiple-wave experiment to estimate welfare-maximizing treatment rules. We provide strong theoretical guarantees and an implementation in a large-scale field experiment.

Policy design in experiments with unknown interference

TL;DR

This paper develops experimental designs for policy design in settings with unknown spillovers confined to a small number of large clusters. It introduces a single-wave approach that leverages local perturbations across two clusters to identify the marginal policy effect (MPE) and to test welfare-optimality, while also enabling estimates of direct, spillover, and welfare effects. It further proposes a multi-wave adaptive design that iteratively learns welfare-maximizing treatment rules with formal guarantees on out-of-sample and in-sample regret, leveraging sequential gradient updates and sequential cross-fitting to handle unobserved interference. The authors provide rigorous theoretical results, including consistency, inference, and fast convergence rates, and implement the methodology in a large-scale field experiment in rural Pakistan (weather-forecast diffusion) with over 250,000 farmers, illustrating substantial welfare gains and practical cost savings. The work contributes to the literature on designing policies under network spillovers and partial interference, offering tractable tools for policy designers when network data are costly or unavailable.

Abstract

This paper studies experimental designs for estimation and inference on policies with spillover effects. Units are organized into a finite number of large clusters and interact in unknown ways within each cluster. First, we introduce a single-wave experiment that, by varying the randomization across cluster pairs, estimates the marginal effect of a change in treatment probabilities, taking spillover effects into account. Using the marginal effect, we propose a test for policy optimality. Second, we design a multiple-wave experiment to estimate welfare-maximizing treatment rules. We provide strong theoretical guarantees and an implementation in a large-scale field experiment.

Paper Structure

This paper contains 120 sections, 37 theorems, 126 equations, 20 figures, 20 tables, 7 algorithms.

Table of Contents

  1. Introduction
  2. Setup and overview
  3. Outcomes, policy choice and welfare maximization
  4. Method's overview: What can we learn with a few clusters?
  5. Single wave
  6. Multi-wave
  7. Main assumptions and applicability of the method
  8. Micro-foundation with network spillovers
  9. Experimental designs
  10. Single-wave experiment: estimation and inference
  11. Algorithm description
  12. Null hypothesis and inference
  13. Other effects identified by the experiment
  14. Multi-wave experiment: welfare maximization
  15. Theoretical guarantees
  16. ...and 105 more sections

Key Result

Proposition 2.1

Consider treatments assigned as in Assumption defn:bernoulli. Let (A) and (B) in Example exmp:microfoundation hold. Then Assumption ass:ass_0 holds.

Figures (20)

  • Figure 1: Example of experimental design, fixing the overall fraction of treated population to be half, and choosing between two types of individuals to treat (those in remote and non-remote regions). The left panel is a single-wave experiment with two clusters. In the first cluster, we assign the policy colored in green, and the second cluster colored in brown. The right panel is a two-wave experiment. We use a pair of clusters to estimate the marginal effect and update the policy for a different pair.
  • Figure 2: Example of the network formation model, with $\gamma_N = 5$. Individuals are assigned different types, which may or may not be observed by the researcher (corresponding to different colors). Individuals interact based on their types and form links among the possible connections. The possible connections and the realized adjacency matrix remain unobserved.
  • Figure 3: The left panel shows the dependence structure when a static policy is implemented on a new population (I omit $D_{i,t-1}^{(k)}$ for expositional convenience), where $\nu_{i,t}$ denote unobservable characteristics. The right panel shows the dependence structure of a sequential experiment that uses the same units for policy updates over subsequent periods with repeated sampling.
  • Figure 4: Sequential cross-fitting method. Clusters (rectangles) are paired. Within each pair, researchers assign different treatment probabilities to clusters with different colors. Finally, the policy in each pair is updated using information from the consecutive pair. Note that because $K > 2 T$, the algorithm never "circles back" to the initial pair.
  • Figure 5: Pakistan's map, organized in districts (each district contains multiple tehsils). Gray regions indicate areas selected for the experiment. Next to each district, we report the total sample size obtained from the tehsils in the experiment in the given district.
  • ...and 15 more figures

Theorems & Definitions (73)

  • Definition 2.1: Welfare
  • Example 2.1: Positive externalities with decreasing returns from neighbors' treatments
  • Example 2.2: Negative externalities
  • Remark 1: Non-separable fixed effects
  • Remark 2: Dependent clusters
  • Remark 3: Global interference
  • Remark 4: Single wave: A free lunch
  • Remark 5: Multiple waves: Alternatives for policy choice
  • Remark 6: Super-population
  • Example 2.3: Microfoundation with network model
  • ...and 63 more