Making FETCH! Happen: Finding Emergent Dog Whistles Through Common Habitats

TL;DR

This work introduces FETCH! as a benchmark for discovering emergent dog whistles in large social-media corpora and presents EarShot, a three-stage system that leverages sentence embeddings, vector databases, and selective LLM/BERT-based filtering to identify novel dog whistles. Across synthetic, balanced, and realistic data scenarios, state-of-the-art methods underperform dramatically (F_{0.5} < 0.05) while EarShot achieves notable gains, especially when using the PREDICT pipeline to maximize precision. The study provides a rigorous evaluation framework, analyzes the strengths and limitations of embedding-based, MLM-based, and hybrid approaches, and discusses practical and ethical implications for deployment in moderation and research. It also outlines future directions, such as hybridizing models and incorporating richer linguistic signals, to improve robustness to recency and context in dog whistle discovery.

Abstract

WARNING: This paper contains content that maybe upsetting or offensive to some readers. Dog whistles are coded expressions with dual meanings: one intended for the general public (outgroup) and another that conveys a specific message to an intended audience (ingroup). Often, these expressions are used to convey controversial political opinions while maintaining plausible deniability and slip by content moderation filters. Identification of dog whistles relies on curated lexicons, which have trouble keeping up to date. We introduce FETCH!, a task for finding novel dog whistles in massive social media corpora. We find that state-of-the-art systems fail to achieve meaningful results across three distinct social media case studies. We present EarShot, a strong baseline system that combines the strengths of vector databases and Large Language Models (LLMs) to efficiently and effectively identify new dog whistles.

Figures (8)

• Figure 1: Comparison between dog whistle detection and our method of dog whistle discovery
• Figure 2: Flowchart for the EarShot System. Two paths can be taken after doing the nearest neighbor lookup. Path (1) is the more computationally expensive system that asks directly prompts a LLM for an answer. Path (2) is the cheaper method that leverages keyword extraction models and binary prediction.
• Figure 3: Word2Vec/Phrase2Vec $F_{0.5}$ performance vs the number of words/phrases returned by the model. Plot is on log scale.
• Figure 4: MLM and EPD $F_{0.5}$ performance vs the prediction threshold. Plot is on log scale.
• Figure 5: $F_{0.5}$ averaged across BERT based filtering methods displaying average performance across keyword extraction models for EarShot BERT PREDICT vs prediction threshold across all three datasets. Plot is on log scale.