Revisiting Noise in Natural Language Processing for Computational Social Science
Nadav Borenstein
TL;DR
This thesis reframes noise in Computational Social Science as a nuanced phenomenon that can encode meaningful social signals, from OCR-induced errors in historical texts to biases and ambiguities in online discourse and LLM outputs. It presents five interconnected case studies: multilingual event extraction from historical newspapers, intersectional bias analysis in OCR-noisy Caribbean corpora, pixel-based language modeling of historical documents, analysis of Schwartz values in Reddit communities, and a tropes-based examination of latent values in large language models. Across these studies, the work demonstrates that simple noise-removal is insufficient; instead, tailored, context-sensitive methods—ranging from extractive QA and cross-lingual transfer to vision-language models and trope clustering—are needed to extract reliable insights from noisy CSS data. The contributions include practical pipelines for historical NLP, robust evaluation frameworks for bias and values, and novel methods to interpret LLM outputs through justification tropes, collectively advancing both methodological rigor and applicability of CSS research in real-world, noise-prone data. The results underscore the importance of nuance in handling noise and highlight how noise-aware approaches can yield deeper, more trustworthy social-science insights with implications for historians, social scientists, and AI practitioners alike.
Abstract
Computational Social Science (CSS) is an emerging field driven by the unprecedented availability of human-generated content for researchers. This field, however, presents a unique set of challenges due to the nature of the theories and datasets it explores, including highly subjective tasks and complex, unstructured textual corpora. Among these challenges, one of the less well-studied topics is the pervasive presence of noise. This thesis aims to address this gap in the literature by presenting a series of interconnected case studies that examine different manifestations of noise in CSS. These include character-level errors following the OCR processing of historical records, archaic language, inconsistencies in annotations for subjective and ambiguous tasks, and even noise and biases introduced by large language models during content generation. This thesis challenges the conventional notion that noise in CSS is inherently harmful or useless. Rather, it argues that certain forms of noise can encode meaningful information that is invaluable for advancing CSS research, such as the unique communication styles of individuals or the culture-dependent nature of datasets and tasks. Further, this thesis highlights the importance of nuance in dealing with noise and the considerations CSS researchers must address when encountering it, demonstrating that different types of noise require distinct strategies.