Benefits and costs of matching prior to a Difference in Differences analysis when parallel trends does not hold

Abstract

The Difference in Difference (DiD) estimator is a popular estimator built on the "parallel trends" assumption, which is an assertion that the treatment group, absent treatment, would change "similarly" to the control group over time. To bolster such a claim, one might generate a comparison group, via matching, that is similar to the treated group with respect to pre-treatment outcomes and/or pre-treatment covariates. Unfortunately, as has been previously pointed out, this intuitively appealing approach also has a cost in terms of bias. To assess the trade-offs of matching in our application, we first characterize the bias of matching prior to a DiD analysis under a linear structural model that allows for time-invariant observed and unobserved confounders with time-varying effects on the outcome. Given our framework, we verify that matching on baseline covariates generally reduces bias. We further show how additionally matching on pre-treatment outcomes has both cost and benefit. First, matching on pre-treatment outcomes partially balances unobserved confounders, which mitigates some bias. This reduction is proportional to the outcome's reliability, a measure of how coupled the outcomes are with the latent covariates. Offsetting these gains, matching also injects bias into the final estimate by undermining the second difference in the DiD via a regression-to-the-mean effect. Consequently, we provide heuristic guidelines for determining to what degree the bias reduction of matching is likely to outweigh the bias cost. We illustrate our guidelines by reanalyzing a principal turnover study that used matching prior to a DiD analysis and find that matching on both the pre-treatment outcomes and observed covariates makes the estimated treatment effect more credible.

Figures (6)

• Figure 1: This event-study style plot shows the estimated margins for treatment and comparison schools relative to year of a principal change (Year 0) for Missouri. The estimates are obtained via a DiD after obtaining a comparable control group through matching on both covariates and lagged outcomes.
• Figure 2: Values of the breakage in parallel trends ($s$, on $x$-axis) and reliability ($r_{\theta}$, on $y$-axis) such that matching on pre-treatment outcomes before DiD leads to less bias (darker areas) or not (lighter areas). See Lemma \ref{['lemma:suffcond_match']}.
• Figure 3: Bias of the naïve DiD estimator (dotted line), matching on $X$ DiD estimator (dashed line), and matching on both $X$ and the pre-treatment outcome (solid line) according to the results in Theorem \ref{['theorem:mainresults']}. We initially fix $\beta_{\theta, 1} = \beta_{x, 1} = 1.5$, $\beta_{\theta, 0} = \beta_{x, 0} = \delta_{\theta} = \delta_x = \sigma_{\theta} = \sigma_x = 1$, $r_{\theta \mid x} = 0.5$, and $\rho = 0$. We then vary one of the fixed parameters for each of the three panels in Figure \ref{['fig:bias_complications']}.
• Figure 4: Guideline \ref{['guideline:matchboth']} performance under model misspecification: all parameters match those in the model presented on Figure \ref{['fig:bias_complications']} of the main paper except we violate Assumption \ref{['assumption:multiple_time_periods']} with two observed covariates Red dotted line represents the turning point on when one should match or not. We have 4 pre-period time points with $\beta_{\theta, t} = 0.0, 0.2, 0.4, 0.6, \beta_{\theta, T} = 0.8$ (a violation of the stability of $\theta$ slopes). We also break stable trends for $\mathbf{X}$ similarly. Left plot shows if we naively followed Guideline \ref{['guideline:matchboth']} (pretending Lemma \ref{['lemma:general_suffcond_match']} holds without Assumption \ref{['assumption:multiple_time_periods']} and also pretending $\beta_{\theta, T-1} = \beta_{\theta, T-2}$ is true) we still correctly classify when to match or not mostly all the time except when near the decision boundary in red. Right plot shows both the true difference in bias of matching only on $\mathbf{X}$ and matching on both observed covariates and pre-period outcome DiD (in green) and the estimated difference, $\hat{\Delta}_{\tau_{x, y}}$, obtained from Guideline \ref{['guideline:matchboth']} (in blue).
• Figure 5: True bias reduction and match recommendation across multifactor simulation scenarios. The columns correspond to the correlation structure of the covariates, with all correlated, the observed correlated with $\theta$ but not themselves, the observed not correlated with $\theta$ but themselves, and all independent. The rows correspond to how $\beta_\theta$ varies: stable, stable in the last two time periods, and fully varying. The final factor of varying coefficients for the observed covariates, or not, does not change the simulation results.

Theorems & Definitions (28)

• Remark 3.1: Time varying confounders
• Remark 3.2: Alternate assignment mechanisms
• Theorem 4.1: Bias of DiD and Matching DiD estimators
• Definition 4.1: Reliability
• Corollary 4.1.1: Bias with no covariates
• Lemma 4.2: Sufficient and necessary condition to match on pre-treatment outcome in the no-covariate case
• Corollary 4.2.1: Bias with zero correlation
• Theorem 5.1: Bias under multiple time periods
• Theorem 5.2: Bias
• Lemma 5.3: Sufficient and necessary condition to match on $T$ pre-treatment outcomes