Measuring Social Media Polarization Using Large Language Models and Heuristic Rules

TL;DR

This work introduces a hybrid framework that uses open-source LLMs (LLaMA $3.1$ $70$B) to annotate social-media content for stance, affect, and agreement, coupled with domain-informed heuristic rules to quantify affective polarization. By applying this approach to climate change and gun control discussions, the authors demonstrate event-driven polarization patterns: anticipatory polarization before well-publicized events and reactive polarization after sudden incidents, with polarization scores ranging from $0$ (constructive) to $10$ (polarizing) and averaged across interactions for longer conversations. The methodology enables scalable, interpretable analysis even in single-interaction threads and leverages a large Twitter/X dataset with time-segmented windows Before, During, and After. The public codebase and results provide a practical pathway for policymakers and researchers to monitor and understand polarization dynamics in online discourse, with potential extensions to broader topics and platforms.

Abstract

Understanding affective polarization in online discourse is crucial for evaluating the societal impact of social media interactions. This study presents a novel framework that leverages large language models (LLMs) and domain-informed heuristics to systematically analyze and quantify affective polarization in discussions on divisive topics such as climate change and gun control. Unlike most prior approaches that relied on sentiment analysis or predefined classifiers, our method integrates LLMs to extract stance, affective tone, and agreement patterns from large-scale social media discussions. We then apply a rule-based scoring system capable of quantifying affective polarization even in small conversations consisting of single interactions, based on stance alignment, emotional content, and interaction dynamics. Our analysis reveals distinct polarization patterns that are event dependent: (i) anticipation-driven polarization, where extreme polarization escalates before well-publicized events, and (ii) reactive polarization, where intense affective polarization spikes immediately after sudden, high-impact events. By combining AI-driven content annotation with domain-informed scoring, our framework offers a scalable and interpretable approach to measuring affective polarization. The source code is publicly available at: https://github.com/hasanjawad001/llm-social-media-polarization.

Figures (5)

• Figure 1: Detailed workflow pipeline of our methodology, illustrating data collection and filtering, LLM-based annotation, application of heuristic rules, and aggregation for polarization insight generation.
• Figure 2: Affective polarization scores across different timeframes (before, during, and after) for four climate change-related events. The bar plots illustrate the average polarization score, with error bars representing the standard error of the mean. The variable $n$ in each bar denotes the number of conversations analyzed during the corresponding timeframe for each event.
• Figure 3: Percentage of conversations exhibiting extreme polarization scores ($x > 7$) across different timeframes (before, during, and after) for climate change-related events.
• Figure 4: Affective polarization scores across different timeframes (before, during, and after) for four gun control-related events. The bar plots display the average polarization score, with error bars indicating the standard error of the mean. The variable $n$ in each bar represents the number of conversations analyzed during the corresponding timeframe for each event.
• Figure 5: Percentage of conversations exhibiting extreme polarization scores ($x > 7$) across different timeframes (before, during, and after) for gun control-related events.