Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Testing Monotonicity in a Finite Population

Jiafeng Chen, Jonathan Roth, Jann Spiess

TL;DR

This paper analyzes monotonicity of treatment effects in a completely randomized finite-population experiment, contrasting the classical superpopulation view with a design-based view. It shows that while the finite-population type counts $\theta$ are identified via the distribution $f_\theta$ of observable data, learning about monotonicity from a single realization is limited: frequentist tests have only modest power and certain priors yield no updating about monotonicity. The authors derive power bounds, demonstrate near-trivial weighted-average power for testing monotonicity, and provide a numerical illustration to illustrate the limits of test-based learning. They also show that some Bayesians update about monotonicity while others do not, depending on the prior, underscoring the role of prior choice in design-based inference. Overall, the work highlights a gap between formal identification and practical inference in design-based causal analysis and suggests careful consideration of how learning is assessed in such settings.

Abstract

We consider the extent to which we can learn from a completely randomized experiment whether all individuals have treatment effects that are weakly of the same sign, a condition we call monotonicity. From a classical sampling perspective, it is well-known that monotonicity is not falsifiable. By contrast, we show from the design-based perspective -- in which the units in the population are fixed and only treatment assignment is stochastic -- that the distribution of treatment effects in the finite population (and hence whether monotonicity holds) is formally identified. We argue, however, that the usual definition of identification is unnatural in the design-based setting because it imagines knowing the distribution of outcomes over different treatment assignments for the same units. We thus evaluate the informativeness of the data by the extent to which it enables frequentist testing and Bayesian updating. We show that frequentist tests can have nontrivial power against some alternatives, but power is generically limited. Likewise, we show that there exist (non-degenerate) Bayesian priors that never update about whether monotonicity holds. We conclude that, despite the formal identification result, the ability to learn about monotonicity from data in practice is severely limited.

Testing Monotonicity in a Finite Population

TL;DR

This paper analyzes monotonicity of treatment effects in a completely randomized finite-population experiment, contrasting the classical superpopulation view with a design-based view. It shows that while the finite-population type counts are identified via the distribution of observable data, learning about monotonicity from a single realization is limited: frequentist tests have only modest power and certain priors yield no updating about monotonicity. The authors derive power bounds, demonstrate near-trivial weighted-average power for testing monotonicity, and provide a numerical illustration to illustrate the limits of test-based learning. They also show that some Bayesians update about monotonicity while others do not, depending on the prior, underscoring the role of prior choice in design-based inference. Overall, the work highlights a gap between formal identification and practical inference in design-based causal analysis and suggests careful consideration of how learning is assessed in such settings.

Abstract

We consider the extent to which we can learn from a completely randomized experiment whether all individuals have treatment effects that are weakly of the same sign, a condition we call monotonicity. From a classical sampling perspective, it is well-known that monotonicity is not falsifiable. By contrast, we show from the design-based perspective -- in which the units in the population are fixed and only treatment assignment is stochastic -- that the distribution of treatment effects in the finite population (and hence whether monotonicity holds) is formally identified. We argue, however, that the usual definition of identification is unnatural in the design-based setting because it imagines knowing the distribution of outcomes over different treatment assignments for the same units. We thus evaluate the informativeness of the data by the extent to which it enables frequentist testing and Bayesian updating. We show that frequentist tests can have nontrivial power against some alternatives, but power is generically limited. Likewise, we show that there exist (non-degenerate) Bayesian priors that never update about whether monotonicity holds. We conclude that, despite the formal identification result, the ability to learn about monotonicity from data in practice is severely limited.
Paper Structure (11 sections, 20 theorems, 81 equations, 2 figures)

This paper contains 11 sections, 20 theorems, 81 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Related literature.
  3. Setup
  4. Identification
  5. Frequentist testing
  6. Intuition for tests.
  7. Numerical illustration with $n=30$.
  8. Unbiased tests.
  9. Bayesian updating
  10. Conclusion
  11. Proofs

Key Result

Proposition 3.1

If $\theta_1 \neq \theta_0 \in \Theta$, then $f_{\theta_1} \neq f_{\theta_0}$, and hence the finite-population type counts $\theta$ are identified. On the other hand, given any $p_1 \in \mathcal{P}_1$, there exists a $p_0 \in \mathcal{P}_0$ such that $g_{p_1} = g_{p_0}$.

Figures (2)

  • Figure 1: Power of the most powerful test for a given alternative $\theta \in \Theta_1$ for $n=30$, $n_1=15$
  • Figure 2: Illustration of the sets $S_{\tilde{\Theta}_0}$ and $S_{\tilde{\Theta}_1}$ with $n_1 = n_0 = 2$

Theorems & Definitions (36)

  • Remark 1: Practical relevance of each null
  • Proposition 3.1
  • Example 1: Illustrative example
  • Remark 2
  • Proposition 4.1
  • Proposition 4.2
  • Proposition 4.3
  • proof : Notes
  • Proposition 4.4
  • Proposition 4.5
  • ...and 26 more