Valid Inference with Imperfect Synthetic Data
Yewon Byun, Shantanu Gupta, Zachary C. Lipton, Rachel Leah Childers, Bryan Wilder
TL;DR
The paper tackles the problem of drawing valid inferences when incorporating synthetic data generated by foundation models into limited-data analyses. It introduces a hyperparameter-free generalized method of moments estimator, GMM-Synth, that augments the target moments with proxy and fully synthetic data via auxiliary parameters, and uses a two-step GMM for efficiency. It provides consistency and asymptotic normality guarantees, showing that synthetic residuals predictive of real residuals can reduce variance, while uninformative synthetic data does not harm asymptotic efficiency. Empirically, GMM-Synth delivers large MSE improvements and substantial gains in effective sample size across four computational social science tasks and two regression settings, outperforming debiasing-based baselines and demonstrating robustness to weaker models. The framework offers a principled, extensible approach for safely leveraging synthetic data from LLMs to support valid, scalable inference in real-world research pipelines.
Abstract
Predictions and generations from large language models are increasingly being explored as an aid in limited data regimes, such as in computational social science and human subjects research. While prior technical work has mainly explored the potential to use model-predicted labels for unlabeled data in a principled manner, there is increasing interest in using large language models to generate entirely new synthetic samples (e.g., synthetic simulations), such as in responses to surveys. However, it remains unclear by what means practitioners can combine such data with real data and yet produce statistically valid conclusions upon them. In this paper, we introduce a new estimator based on generalized method of moments, providing a hyperparameter-free solution with strong theoretical guarantees to address this challenge. Intriguingly, we find that interactions between the moment residuals of synthetic data and those of real data (i.e., when they are predictive of each other) can greatly improve estimates of the target parameter. We validate the finite-sample performance of our estimator across different tasks in computational social science applications, demonstrating large empirical gains.