Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synthetic Control Misconceptions: Recommendations for Practice

Robert Pickett, Jennifer Hill, Sarah Cowan

Abstract

To estimate the causal effect of an intervention, researchers need to identify a control group that represents what might have happened to the treatment group in the absence of that intervention. This is challenging without a randomized experiment and further complicated when few units (possibly only one) are treated. Nevertheless, when data are available on units over time, synthetic control (SC) methods provide an opportunity to construct a valid comparison by differentially weighting control units that did not receive the treatment so that their resulting pre-treatment trajectory is similar to that of the treated unit. The hope is that this weighted ``pseudo-counterfactual" can serve as a valid counterfactual in the post-treatment time period. Since its origin twenty years ago, SC has been used over 5,000 times in the literature (Web of Science, December 2025), leading to a proliferation of descriptions of the method and guidance on proper usage that is not always accurate and does not always align with what the original developers appear to have intended. As such, a number of accepted pieces of wisdom have arisen: (1) SC is robust to various implementations; (2) covariates are unnecessary, and (3) pre-treatment prediction error should guide model selection. We describe each in detail and conduct simulations that suggest, both for standard and alternative implementations of SC, that these purported truths are not supported by empirical evidence and thus actually represent misconceptions about best practice. Instead of relying on these misconceptions, we offer practical advice for more cautious implementation and interpretation of results.

Synthetic Control Misconceptions: Recommendations for Practice

Abstract

To estimate the causal effect of an intervention, researchers need to identify a control group that represents what might have happened to the treatment group in the absence of that intervention. This is challenging without a randomized experiment and further complicated when few units (possibly only one) are treated. Nevertheless, when data are available on units over time, synthetic control (SC) methods provide an opportunity to construct a valid comparison by differentially weighting control units that did not receive the treatment so that their resulting pre-treatment trajectory is similar to that of the treated unit. The hope is that this weighted ``pseudo-counterfactual" can serve as a valid counterfactual in the post-treatment time period. Since its origin twenty years ago, SC has been used over 5,000 times in the literature (Web of Science, December 2025), leading to a proliferation of descriptions of the method and guidance on proper usage that is not always accurate and does not always align with what the original developers appear to have intended. As such, a number of accepted pieces of wisdom have arisen: (1) SC is robust to various implementations; (2) covariates are unnecessary, and (3) pre-treatment prediction error should guide model selection. We describe each in detail and conduct simulations that suggest, both for standard and alternative implementations of SC, that these purported truths are not supported by empirical evidence and thus actually represent misconceptions about best practice. Instead of relying on these misconceptions, we offer practical advice for more cautious implementation and interpretation of results.
Paper Structure (40 sections, 18 equations, 22 figures)

This paper contains 40 sections, 18 equations, 22 figures.

Table of Contents

  1. Introduction
  2. Empirical Case: Alaska and the Permanent Fund Dividend
  3. Synth and its Implementation
  4. Misconceptions of Synthetic Control
  5. Misconception 1: Synthetic Control is Robust to Implementation Choices
  6. Misconception 1a: Nested vs Regression Weights are Interchangeable
  7. Misconception 1b: Reference Category Choice is Inconsequential
  8. Misconception 2: Covariates are not necessary when using SC
  9. Misconception 3: Lower Pre-Treatment Outcome Imbalance Suggests Lower Absolute Bias
  10. Simulation
  11. Simulation Overview
  12. Simulation Calibration
  13. Simulating Covariates and Inducing Treatment Assignment
  14. Simulating outcomes and treatment effects
  15. Method Details
  16. ...and 25 more sections

Figures (22)

  • Figure 1: Contributions to $\boldsymbol{V}$
  • Figure 2: Simulation Overview
  • Figure 3: Average Root Mean Squared Error of the estimated causal effect by Simulation Scenario, Method, and Outcome Data Generating Model. Vertical jitter added to help distinguish between points representing methods with similar performance.
  • Figure 4: Variation in Synthetic Control Estimates Due To Reference Category Choice by Method and Simulation Scenario. Vertical jitter added to help distinguish between points representing methods with similar performance.
  • Figure 5: Root mean squared error of the estimated causal effect using All or No Covariates by simulation scenario, method, and outcome data generating model. Horizontal jitter added to help distinguish between points representing methods with similar performance.
  • ...and 17 more figures