Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Counterfactual Policies Using Instruments

Michal Kolesár, José Luis Montiel Olea, Jonathan Roth

Abstract

We study settings in which a researcher has an instrumental variable (IV) and seeks to evaluate the effects of a counterfactual policy that alters treatment assignment, such as a directive encouraging randomly assigned judges to release more defendants. We develop a general and computationally tractable framework for computing sharp bounds on the effects of such policies. Our approach does not require the often tenuous IV monotonicity assumption. Moreover, for an important class of policy exercises, we show that IV monotonicity -- while crucial for a causal interpretation of two-stage least squares -- does not tighten the bounds on the counterfactual policy impact. We analyze the identifying power of alternative restrictions, including the policy invariance assumption used in the marginal treatment effect literature, and develop a relaxation of this assumption. We illustrate our framework using applications to quasi-random assignment of bail judges in New York City and prosecutors in Massachusetts.

Evaluating Counterfactual Policies Using Instruments

Abstract

We study settings in which a researcher has an instrumental variable (IV) and seeks to evaluate the effects of a counterfactual policy that alters treatment assignment, such as a directive encouraging randomly assigned judges to release more defendants. We develop a general and computationally tractable framework for computing sharp bounds on the effects of such policies. Our approach does not require the often tenuous IV monotonicity assumption. Moreover, for an important class of policy exercises, we show that IV monotonicity -- while crucial for a causal interpretation of two-stage least squares -- does not tighten the bounds on the counterfactual policy impact. We analyze the identifying power of alternative restrictions, including the policy invariance assumption used in the marginal treatment effect literature, and develop a relaxation of this assumption. We illustrate our framework using applications to quasi-random assignment of bail judges in New York City and prosecutors in Massachusetts.
Paper Structure (51 sections, 9 theorems, 77 equations, 3 figures, 7 tables)

This paper contains 51 sections, 9 theorems, 77 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Setup
  3. Observable data and potential outcomes
  4. The identified set for the average counterfactual outcome
  5. Examples of restrictions on potential treatments
  6. The identified set without monotonicity or policy invariance
  7. Computation via linear programming
  8. Does IV monotonicity help tighten the identified set?
  9. Simple example for intuition.
  10. More general sufficient conditions.
  11. What assumptions help tighten the identified set?
  12. Policy invariance and its relaxations
  13. Example: quota policy.
  14. Relaxing policy invariance with disagreement bounds.
  15. Calculating the identified set under disagreement bounds.
  16. ...and 36 more sections

Key Result

Proposition 1

Suppose that $\mathcal{Y}$ is finite. There exists a joint distribution $P^* \in \mathcal{P}^*_{I}(P;\mathcal{P}^{*}_{\textnormal{valid}})$ with marginals $\{\pi_{z}(\cdot) \}_{z \in \mathcal{Z}}$ if and only if $\{\pi_{z}(\cdot) \}_{z \in \mathcal{Z}}$ satisfy the following three conditions:

Figures (3)

  • Figure 6.1: Judge- and race-specific sample release and misconduct rates, based on NYC bail judge data from adh22.
  • Figure 6.2: Illustration of identified set for $E[Y(1) \mid R=b]$ using NYC bail judge data.
  • Figure 6.3: ADA-specific sample non-prosecution rates and criminal complaint rates, based on Suffolk County prosecutor data from adh23.

Theorems & Definitions (19)

  • Proposition 1
  • Corollary 1
  • Remark 1: Discretizing $Y$
  • Remark 2: Comparing multiple policies
  • Remark 3: Estimation and inference
  • Proposition 2: No identifying power of monotonicity with known $Y(0)$
  • Proposition 3: No identifying power of monotonicity with strong encouragement
  • Remark 4: Interaction of IV monotonicity with other constraints
  • Remark 5: Weaker monotonicity conditions
  • Remark 6: Relationship to literature
  • ...and 9 more