State policy heterogeneity analyses: considerations and proposals
Max Rubinstein, Megan S. Schuler, Elizabeth A. Stuart, Bradley D. Stein, Max Griswold, Elizabeth M. Stone, Beth Ann Griffin
TL;DR
This paper examines why heterogeneity analyses in state-policy research frequently yield associational rather than causal insights and clarifies the distinct causal estimands—ITE, CATE, and CDE—relevant to heterogeneity. It argues that coarsening policies across states further obscures policy-relevant interpretation and proposes bounding state-specific treatment effects (ITE) as a principled alternative, using a DiD framework and pre-treatment data to calibrate sensitivity parameters. Through theory, simulations, and an application to ACA Medicaid expansion’s impact on high-volume buprenorphine prescribing, the authors show that bounds can yield robust, state-specific insights even when point identification of CATE or CDE is weak or infeasible. The work emphasizes reporting clearly defined estimands, conditions for identification, and the interpretability of bounds, offering a practical toolkit for policy-makers and researchers dealing with limited state-level data. Overall, bounding ITEs provides a transparent, policy-relevant approach to quantify heterogeneity in state policy effects under realistic data constraints.
Abstract
State-level policy studies often conduct heterogeneity analyses that quantify how treatment effects vary across state characteristics. These analyses may be used to inform state-specific policy decisions, or to infer how the effect of a policy changes in combination with other state characteristics. However, in state-level settings with varied contexts and policy landscapes, multiple versions of similar policies, and differential policy implementation, the causal quantities targeted by these analyses may not align with the inferential goals. This paper clarifies these issues by distinguishing several causal estimands relevant to heterogeneity analyses in state-policy settings, including state-specific treatment effects (ITE), conditional average treatment effects (CATE), and controlled direct effects (CDE). We argue that the CATE is often the easiest to identify and estimate, but may not be the most policy relevant target of inference. Moreover, the widespread practice of coarsening distinct policies or implementations into a single indicator further complicates the interpretation of these analyses. Motivated by these limitations, we propose bounding ITEs as an alternative inferential goal, yielding ranges for each state's policy effect under explicit assumptions that quantify deviations from the ideal identifying conditions. These bounds target a well-defined and policy-relevant quantity, the effect for specific states. We develop this approach within a difference-in-differences framework and discuss how sensitivity parameters may be informed using pre-treatment data. Through simulations we demonstrate that bounding state-specific effects can more reliably determine the sign of the ITEs than CATE estimates. We then illustrate this method to examine the effect of the Affordable Care Act Medicaid expansion on high-volume buprenorphine prescribing.