Exploring the Capability of ChatGPT to Reproduce Human Labels for Social Computing Tasks (Extended Version)

TL;DR

This work investigates whether ChatGPT can reliably reproduce human annotations for social computing tasks to reduce labeling costs. By re-annotating seven English datasets across diverse social issues, the authors quantify ChatGPT’s performance (average weighted F1 = 72.00%, best on clickbait at ~89.66%) and identify label- and task-specific gaps. To guide researchers, they introduce GPT-Rater, a predictive tool that estimates ChatGPT’s labeling success using document embeddings and multiple classifiers, achieving high accuracy especially on datasets where ChatGPT performs well (e.g., Clickbait Headlines with μ accuracy = 90.59% and μ F1 = 95.00%). The study demonstrates a promising, though uneven, potential for automating data annotation in social computing and provides a practical mechanism to forecast ChatGPT’s usefulness before large-scale annotation. Overall, GPT-Rater enables data-driven decisions about when to rely on ChatGPT for labeling tasks, potentially lowering barriers to social computing research while highlighting domain constraints and the need for task-specific prompting and training.

Abstract

Harnessing the potential of large language models (LLMs) like ChatGPT can help address social challenges through inclusive, ethical, and sustainable means. In this paper, we investigate the extent to which ChatGPT can annotate data for social computing tasks, aiming to reduce the complexity and cost of undertaking web research. To evaluate ChatGPT's potential, we re-annotate seven datasets using ChatGPT, covering topics related to pressing social issues like COVID-19 misinformation, social bot deception, cyberbully, clickbait news, and the Russo-Ukrainian War. Our findings demonstrate that ChatGPT exhibits promise in handling these data annotation tasks, albeit with some challenges. Across the seven datasets, ChatGPT achieves an average annotation F1-score of 72.00%. Its performance excels in clickbait news annotation, correctly labeling 89.66% of the data. However, we also observe significant variations in performance across individual labels. Our study reveals predictable patterns in ChatGPT's annotation performance. Thus, we propose GPT-Rater, a tool to predict if ChatGPT can correctly label data for a given annotation task. Researchers can use this to identify where ChatGPT might be suitable for their annotation requirements. We show that GPT-Rater effectively predicts ChatGPT's performance. It performs best on a clickbait headlines dataset by achieving an average F1-score of 95.00%. We believe that this research opens new avenues for analysis and can reduce barriers to engaging in social computing research.

Figures (2)

• Figure 1: The confusion matrices of ChatGPT's annotations for the seven annotation datasets. The y-axis label for a given row shows human label of text, and values in each cell show the percentage of those text annotated in the corresponding x-axis label by ChatGPT.
• Figure 2: Distribution of F1-scores of GPT-Rater classifiers over subsets by different proportions. The x-axis "Proportion" denotes the proportion in which subsets are sampled from corresponding full datasets to retrain and test GPT-Rater classifiers, within a 80:20 train-test split. A "" denotes the lowest proportion where a GPT-Rater classifier can achieve a F1-score close to its F1-score in corresponding full datasets (gap $<1\%$).