Partial Identification under Missing Data Using Weak Shadow Variables from Pretrained Models
Hongyu Chen, David Simchi-Levi, Ruoxuan Xiong
TL;DR
This work addresses estimating population quantities under missing-not-at-random data by adopting a partial identification framework expressed through a pair of linear programs, yielding sharp bounds on the mean outcome. It introduces weak shadow variables from pretrained models, notably LLMs, as auxiliary predictions that satisfy a conditional independence constraint but need not meet classical completeness, enabling substantial tightening of identification regions. A set-expansion estimator is developed to ensure finite-sample validity and to provide convergence rates that adapt to whether identification is partial or point, with faster rates when the shadow information yields point identification. The approach is extended to randomized experiments and validated via numerical simulations and semi-synthetic analyses on customer-service dialogues, showing that LLM-derived weak shadows shrink bound widths by 75–83% while maintaining valid coverage under MNAR. The results highlight a practical path to leverage rich external models to improve inference under missing data without relying on brittle parametric assumptions or perfect predictive accuracy.
Abstract
Estimating population quantities such as mean outcomes from user feedback is fundamental to platform evaluation and social science, yet feedback is often missing not at random (MNAR): users with stronger opinions are more likely to respond, so standard estimators are biased and the estimand is not identified without additional assumptions. Existing approaches typically rely on strong parametric assumptions or bespoke auxiliary variables that may be unavailable in practice. In this paper, we develop a partial identification framework in which sharp bounds on the estimand are obtained by solving a pair of linear programs whose constraints encode the observed data structure. This formulation naturally incorporates outcome predictions from pretrained models, including large language models (LLMs), as additional linear constraints that tighten the feasible set. We call these predictions weak shadow variables: they satisfy a conditional independence assumption with respect to missingness but need not meet the completeness conditions required by classical shadow-variable methods. When predictions are sufficiently informative, the bounds collapse to a point, recovering standard identification as a special case. In finite samples, to provide valid coverage of the identified set, we propose a set-expansion estimator that achieves slower-than-$\sqrt{n}$ convergence rate in the set-identified regime and the standard $\sqrt{n}$ rate under point identification. In simulations and semi-synthetic experiments on customer-service dialogues, we find that LLM predictions are often ill-conditioned for classical shadow-variable methods yet remain highly effective in our framework. They shrink identification intervals by 75--83\% while maintaining valid coverage under realistic MNAR mechanisms.