Silent Signals, Loud Impact: LLMs for Word-Sense Disambiguation of Coded Dog Whistles

TL;DR

This work tackles the challenge of identifying and disambiguating coded dog whistles in political and social discourse using Large Language Models. It introduces Silent Signals, the largest high-confidence dataset of disambiguated dog whistle usage (16,550 examples across 298 dog whistles) drawn from formal Congressional records and informal Reddit posts, and pairs this with a Potential Instance dataset to enable large-scale study. The study shows LLMs struggle with automatic dog whistle resolution, though GPT-4 with ensemble prompting achieves high precision on disambiguation tasks, enabling the creation of a high-quality resource for hate speech detection, neology tracking, and political science analyses. The Silent Signals dataset and accompanying methodology provide a foundation for analyzing the emergence and evolution of coded language and its impact on online moderation and political discourse.

Abstract

A dog whistle is a form of coded communication that carries a secondary meaning to specific audiences and is often weaponized for racial and socioeconomic discrimination. Dog whistling historically originated from United States politics, but in recent years has taken root in social media as a means of evading hate speech detection systems and maintaining plausible deniability. In this paper, we present an approach for word-sense disambiguation of dog whistles from standard speech using Large Language Models (LLMs), and leverage this technique to create a dataset of 16,550 high-confidence coded examples of dog whistles used in formal and informal communication. Silent Signals is the largest dataset of disambiguated dog whistle usage, created for applications in hate speech detection, neology, and political science. The dataset can be found at https://huggingface.co/datasets/SALT-NLP/silent_signals.

Figures (9)

• Figure 1: This figure demonstrates the nuances of dog whistle detection as a word can be used in a coded or non-coded sense. All illustrations were created using Adobe Firefly.
• Figure 2: Visual representation of the different prompt structures used in Automatic Dog Whistle Resolution (Section \ref{['sec:baseline']}) and Word-Sense Disambiguation (Section \ref{['subsec:wsd']}) experiments.
• Figure 3: Results of Dog Whistle Disambiguation task using the simulated ensemble across $N=1,3,5$ inferences. In an attempt to compensate for output volatility, for each N-inferences experiment, predictions are only considered if they remained consistent across all $N$ runs. Precision-1 and Recall-1 scores pertain to the positive class of coded dog whistle instances.
• Figure 4: The distributions of dog whistles over in-groups for informal and formal communication in the Silent Signals dataset.
• Figure 5: The distributions of dog whistles over time for informal and formal communication in the Silent Signals dataset.