Surrogate Representation Inference for Text and Image Annotations

TL;DR

This work develops Surrogate Representation Inference (SRI), a framework for making valid, efficient inferences when unstructured data (texts/images) are used to predict or annotate structured variables. By learning low-dimensional surrogate representations of unstructured data that fully mediate the relationship between human annotations and outcomes, SRI achieves semiparametric efficiency and reduces standard errors, even when human labels contain non-differential errors. The methodology combines a DragonNet-like neural architecture, efficient influence function theory, and cross-fitting-based estimation, with extensions to multiple coders and noisy annotations. An empirical application to framing of immigrants in Congressional speeches demonstrates substantial variance reduction compared with existing bias-correction approaches, underscoring the practical value of SRI for text- and image-based inference. The paper also discusses diagnostic procedures, assumptions testing via permutation, and potential extensions to broader modalities and data-driven discovery of outcome concepts.

Abstract

As researchers increasingly rely on machine learning models and LLMs to annotate unstructured data, such as texts or images, various approaches have been proposed to correct bias in downstream statistical analysis. However, existing methods tend to yield large standard errors and require some error-free human annotation. In this paper, I introduce Surrogate Representation Inference (SRI), which assumes that unstructured data fully mediate the relationship between human annotations and structured variables. The assumption is guaranteed by design provided that human coders rely only on unstructured data for annotation. Under this setting, I propose a neural network architecture that learns a low-dimensional representation of unstructured data such that the surrogate assumption remains to be satisfied. When multiple human annotations are available, SRI can be extended to further correct non-differential measurement errors that may exist in human annotations. Focusing on text-as-outcome settings, I formally establish the identification conditions and semiparametric efficient estimation strategies that enable learning and leveraging such a low-dimensional representation. Simulation studies and a real-world application demonstrate that SRI reduces standard errors by over 50% when machine learning classification accuracy is moderate and provides valid inference even when human annotations contain non-differential measurement errors.

Figures (7)

• Figure 1: The diagram illustrating the assumptions for the proposed method. $T$ denotes the predictors of interest, $\bm{Y}$ denotes the texts (outcomes), $\bm{Z}$ denotes the set of all control variables, and $\tilde{L}$ denotes the human annotated labels. An arrow with red double lines represents a deterministic relation while an arrow with a single line indicates a possibly stochastic relationship. It is assumed that when coders annotate $\tilde{L}$, they only look at texts $\bm{Y}$ and do not look at the predictors of interest $T$ directly. Under this setup, texts $\bm{Y}$ can be used as a surrogate variable that fully mediates $T$'s effect on $\tilde{L}$.
• Figure 2: The architecture of neural network for the nuisance function estimation. Given the inputs $\bm{Y}_i$ (texts) and $\bm{Z}_i$ (covariates), the network maps them onto the shared internal representation $\bm{W}_i = \bm{f}(\bm{Y}_i, \bm{Z}_i; \xi)$ parametrized by $\xi$. This representation $\bm{W}_i$ is used to simultaneously predict the human annotation $\tilde{L}_i$ (outcome model $\mathbb{E}[\tilde{L}_i \mid \bm{W}_i]$) and the predictor of interest $T_i$ (surrogacy score $\mathbb{P}(T_i = t \mid \bm{W}_i)$).
• Figure 3: The diagram illustrates the assumptions for the proposed method when human annotations may differ from the unobserved true label of interest. $T$ denotes the predictors of interest, $\bm{Y}$ denotes the texts (outcomes), $L$ represents the unobserved true concept of interest, and $\bm{Z}$ denotes the set of control variables. $\tilde{L}^{(j)}$ indicates the human-annotated labels provided by coder $j$ ($j = 1,2$). An arrow with red double lines represents a deterministic relationship, while an arrow with a single line indicates a potentially stochastic relationship. White nodes correspond to observed variables, whereas gray nodes indicate unobserved variables.
• Figure 4: The estimated partisan difference in tone towards immigrants based on the data of card_computational_2022 for (1) the SRI estimator (proposed), (2) the existing bias-correction approach using both human annotations and machine learning predictions; angelopoulos2023predictionegami_neulips_2023egami2024using), and (3) the naïve estimator (directly using the machine learning predictions).
• Figure S1: Workflow for the applied researchers to create the coding rule and the labeled data with multiple human coders. To test the conditional independence assumption (Assumption \ref{['proximal']}) in Step 3, the conditional permutation test is used. See the main text for the specific procedure.

Theorems & Definitions (13)

• Definition 1: Surrogate Representation of Texts
• Lemma 1: Existence of Surrogate Representation
• Proposition 1: Identification under Perfect Annotations
• Theorem 1: Efficient Influence Function
• Theorem 2: Asymptotic Normality
• Proposition 2: Identification without Perfect Human Annotations
• Theorem 3: Influence Function
• Theorem 4: Asymptotic Normality
• Theorem 5: Asymptotic Normality with Augmented Outcome
• Remark 1