Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models

Naoki Egami, Musashi Hinck, Brandon M. Stewart, Hanying Wei

TL;DR

<3-5 sentence high-level summary> Social science analyses often rely on labeled documents to explain social phenomena, but using imperfect LLM surrogates for downstream regression biases inference. The authors introduce design-based supervised learning (DSL), a doubly robust method that combines surrogate labels with a small gold-standard annotated set to produce bias-corrected pseudo-outcomes and valid inference under known sampling probabilities. They prove consistency and asymptotic normality and extend the method to moment-based estimators, with empirical validation across 18 CSS datasets showing competitive RMSE and reliable coverage, outperforming surrogate-only and gold-standard-only approaches in bias and uncertainty quantification. The work provides a practical, theoretically grounded approach for leveraging LLM annotations in social science research without sacrificing inferential validity.

Abstract

In computational social science (CSS), researchers analyze documents to explain social and political phenomena. In most scenarios, CSS researchers first obtain labels for documents and then explain labels using interpretable regression analyses in the second step. One increasingly common way to annotate documents cheaply at scale is through large language models (LLMs). However, like other scalable ways of producing annotations, such surrogate labels are often imperfect and biased. We present a new algorithm for using imperfect annotation surrogates for downstream statistical analyses while guaranteeing statistical properties -- like asymptotic unbiasedness and proper uncertainty quantification -- which are fundamental to CSS research. We show that direct use of surrogate labels in downstream statistical analyses leads to substantial bias and invalid confidence intervals, even with high surrogate accuracy of 80-90%. To address this, we build on debiased machine learning to propose the design-based supervised learning (DSL) estimator. DSL employs a doubly-robust procedure to combine surrogate labels with a smaller number of high-quality, gold-standard labels. Our approach guarantees valid inference for downstream statistical analyses, even when surrogates are arbitrarily biased and without requiring stringent assumptions, by controlling the probability of sampling documents for gold-standard labeling. Both our theoretical analysis and experimental results show that DSL provides valid statistical inference while achieving root mean squared errors comparable to existing alternatives that focus only on prediction without inferential guarantees.

Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models

TL;DR

<3-5 sentence high-level summary> Social science analyses often rely on labeled documents to explain social phenomena, but using imperfect LLM surrogates for downstream regression biases inference. The authors introduce design-based supervised learning (DSL), a doubly robust method that combines surrogate labels with a small gold-standard annotated set to produce bias-corrected pseudo-outcomes and valid inference under known sampling probabilities. They prove consistency and asymptotic normality and extend the method to moment-based estimators, with empirical validation across 18 CSS datasets showing competitive RMSE and reliable coverage, outperforming surrogate-only and gold-standard-only approaches in bias and uncertainty quantification. The work provides a practical, theoretically grounded approach for leveraging LLM annotations in social science research without sacrificing inferential validity.

Abstract

In computational social science (CSS), researchers analyze documents to explain social and political phenomena. In most scenarios, CSS researchers first obtain labels for documents and then explain labels using interpretable regression analyses in the second step. One increasingly common way to annotate documents cheaply at scale is through large language models (LLMs). However, like other scalable ways of producing annotations, such surrogate labels are often imperfect and biased. We present a new algorithm for using imperfect annotation surrogates for downstream statistical analyses while guaranteeing statistical properties -- like asymptotic unbiasedness and proper uncertainty quantification -- which are fundamental to CSS research. We show that direct use of surrogate labels in downstream statistical analyses leads to substantial bias and invalid confidence intervals, even with high surrogate accuracy of 80-90%. To address this, we build on debiased machine learning to propose the design-based supervised learning (DSL) estimator. DSL employs a doubly-robust procedure to combine surrogate labels with a smaller number of high-quality, gold-standard labels. Our approach guarantees valid inference for downstream statistical analyses, even when surrogates are arbitrarily biased and without requiring stringent assumptions, by controlling the probability of sampling documents for gold-standard labeling. Both our theoretical analysis and experimental results show that DSL provides valid statistical inference while achieving root mean squared errors comparable to existing alternatives that focus only on prediction without inferential guarantees.
Paper Structure (17 sections, 2 theorems, 9 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 17 sections, 2 theorems, 9 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work.
  4. The Problem Setting and Design-based Sampling
  5. Existing Approaches: Their Methodological Challenges
  6. Surrogate Only Estimation (SO)
  7. Gold-Standard Only Estimation (GSO)
  8. Supervised Learning (SL)
  9. Our Proposed Estimator: Design-based Supervised Learning
  10. Extension: Method of Moment Estimator
  11. Experiments
  12. Logistic Regression in Congressional Bills Data
  13. Class Prevalence Estimation
  14. Discussion
  15. Limitations.
  16. ...and 2 more sections

Key Result

Proposition 1

Under Assumption assum-1, when the DSL estimator is fitted with the cross-fitting approach (Algorithm alg:dsl), estimated coefficients $\widehat{\beta}$ are consistent and asymptotically normal. In addition, the following variance estimator $\widehat{V}$is consistent to $V$, that is, $\widehat{V} \xrightarrow[]{p} V$, where

Figures (3)

  • Figure 1: Overview of the Problem and Design-based Supervised Learning (DSL)
  • Figure 2: Logistic regression estimation with Congressional Bills Project Data. Results for a three variable logistic regression model of a binary outcome indicating whether a bill is about Macroeconomy. Bias shows the standardized root mean squared bias (averaged over the three coefficients). Coverage shows proportion of $95\%$ confidence intervals covering the truth. RMSE plots the average RMSE of the coefficients on a log scale. Each sampled dataset contains 10K datapoints with the X-axis providing gold-standard sample size. We average over 500 simulations at each point. Only DSL and GSO are able to achieve proper coverage, but DSL is more efficient.
  • Figure 3: Improvement of DSL over GSO. Both DSL and GSO attain asymptotic unbiasedness and proper coverage. Here we show the gain in efficiency for DSL over GSO in the balanced condition. As the quality of the surrogate rises (here as we move from the 0-shot to 5-shot setting) the efficiency gain from DSL grows.

Theorems & Definitions (4)

  • Definition 1: Design-based Supervised Learning Estimator
  • Proposition 1
  • Definition 2: Design-based Moments
  • Proposition 2