From Measurement Instruments to Data: Leveraging Theory-Driven Synthetic Training Data for Classifying Social Constructs

TL;DR

This work tackles the challenge of scarce high-quality labeled data for social science text classification by introducing theory-driven synthetic training data generated from established measurement instruments such as survey scales and annotation codebooks. Through two case studies on sexism and political topics, the authors evaluate how theory-guided prompts and data generation strategies influence in-domain and out-of-domain classification performance using RoBERTa and SVM baselines, with GPT-4o as a direct labeling benchmark. The findings are mixed: theory-driven synthetic data meaningfully enhances political topic classification and can reduce labeling needs, but shows limited or negative impact for sexism, suggesting the utility of instrument choice and task framing. The study highlights the potential and limits of incorporating social science theory into synthetic data generation and encourages further exploration across more constructs, instruments, and model architectures to improve measurement in computational social science.

Abstract

Computational text classification is a challenging task, especially for multi-dimensional social constructs. Recently, there has been increasing discussion that synthetic training data could enhance classification by offering examples of how these constructs are represented in texts. In this paper, we systematically examine the potential of theory-driven synthetic training data for improving the measurement of social constructs. In particular, we explore how researchers can transfer established knowledge from measurement instruments in the social sciences, such as survey scales or annotation codebooks, into theory-driven generation of synthetic data. Using two studies on measuring sexism and political topics, we assess the added value of synthetic training data for fine-tuning text classification models. Although the results of the sexism study were less promising, our findings demonstrate that synthetic data can be highly effective in reducing the need for labeled data in political topic classification. With only a minimal drop in performance, synthetic data allows for substituting large amounts of labeled data. Furthermore, theory-driven synthetic data performed markedly better than data generated without conceptual information in mind.

Figures (4)

• Figure 1: Computational workflow to generate training datasets that contain different shares of synthetic and human-labeled data
• Figure 2: Performance on In-Domain Test Set. The y-axis depicts the macro-F1 score across three random seeds. Error bars represent 95% confidence intervals across the three training runs. Training sizes were balanced across classes, with $N_{Training}$ = 1,938 for the sexism study (969 cases per class) and $N_{Training}$ = 3,500 for the topics study (500 cases per class). The dark grey lines correspond to the results of a model trained on only the labeled subset of the data; synthetic data was generated using GPT-3.5-turbo.
• Figure 4: Performance on OOD Set. The y-axis depicts the macro-F1 score across three random seeds. Error bars represent 95% confidence intervals across the three training runs. Training sizes were balanced across classes, with $N_{Training}$ = 1,938 for the sexism study (969 cases per class) and $N_{Training}$ = 3,500 for the topics study (500 cases per class). The dark grey lines correspond to the results of a model trained on only the labeled subset of the data; synthetic data was generated using GPT-3.5-turbo.
• Figure 5: Performance and Model Generation.The y-axis corresponds to the mean performance across data generation strategies and random seeds for each share of synthetic data and generation model. The dark grey lines correspond to the baseline model that was only trained on labeled data.