Quantifying Polarization: A Comparative Study of Measures and Methods

TL;DR

This paper tackles how to quantify political polarization in ideological distributions by comparing five widely used measures on synthetic benchmarks and a YouTube case study of 2020 U.S. election discussions. It introduces an adaptation of Kleinberg's burst-detection algorithm to count distribution modes and pairs it with existing metrics to improve mode detection and interpretability. Results reveal no single metric is universally best; BC can misclassify skewed unimodal distributions, A depends on mode spacing, and DFU/Dip provide different signals that align better with observed multimodality when augmented by burst-based mode counts. The work provides a practical framework for analyzing online polarization and a methodological tool for more nuanced interpretation of ideological patterns in social media, with center-leaning content often showing greater cross-talk.

Abstract

Political polarization, a key driver of social fragmentation, has drawn increasing attention for its role in shaping online and offline discourse. Despite significant efforts, accurately measuring polarization within ideological distributions remains a challenge. This study evaluates five widely used polarization measures, testing their strengths and weaknesses with synthetic datasets and a real-world case study on YouTube discussions during the 2020 U.S. Presidential Election. Building on these findings, we present a novel adaptation of Kleinberg's burst detection algorithm to improve mode detection in polarized distributions. By offering both a critical review and an innovative methodological tool, this work advances the analysis of ideological patterns in social media discourse.

Figures (6)

• Figure 1: Measures' performance on synthetically generated distributions. The tables above each panel show the score for the different measures. All of the distributions are in the interval $[-1, +1]$. An estimation of the density has been added on top of the histograms for ease of visualization.
• Figure 2: Empirical Complementary Cumulative Distribution Function (ECCDF) of the number of comments per user and density of the inferred users' leaning. Most of the users in the data set post few comments, while only a minority are relatively active: the median number of comments per user is $2$ (mean = $8.7$), and $48\%$ of the user base in the data set only posted one comment. The users' leaning instead is skewed towards left-leaning values. The scores of the measures computed on the inferred user leanings paint a polarized scenario.
• Figure 3: ECCDF of the polarization measures' scores over the leaning array of the comment sections (Panel A). The vertical line at $x=0.55$ indicates the benchmark threshold for a bimodal distribution when considering the Bimodality Coefficient. The boxplot of the distribution of the different measures differentiated by leaning (Panel B) shows us how all of the measures - bar the Bimodality Coefficient - consider center-leaning comment sections as more polarized than their left and right-leaning counterparts. The DFU scores in both panels are rescaled to the interval $[0, 1]$ to ensure visual consistency with the other measures.
• Figure 4: Results of Kleinberg's burst detection algorithm, toy example. Panel A refers to the toy distribution, while Panel B illustrates the burst detection output. The algorithm detects different peaks of varying intensity coinciding with the distributions' modes. The level of intensity 1 always spans the entire sample range.
• Figure 5: Results of Kleinberg's burst detection algorithm, and the subsequent burst-aggregating procedure, on different leanings' distributions. The burst-aggregating procedure are presented for two different levels of $\alpha$ and $k$. In Panel A, the different values of $\alpha$ and $k$ produce substantially different outputs (identifying, respectively, two and four bursts of intensity higher than 3), while in Panel B there is total agreement regarding the burst aggregating procedure.