Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Faces Speak Louder Than Words: Emotions Versus Textual Sentiment in the 2024 USA Presidential Election

Chiyu Wei, Sean Noh, Ho-Chun Herbert Chang

TL;DR

The paper addresses how emotional content expressed in images (facial expressions) relates to textual sentiment in social media during the 2024 U.S. presidential election. It employs a multimodal pipeline that collects Instagram posts (CrowdTangle data), applies TextBlob, VADER, and DistilBERT for text, and uses Py-Feat to extract facial Action Units and classify emotions, integrating per-face analyses and a weighted cross-modal score. Key findings show that facial emotion often diverges from caption sentiment yet aligns in systemic patterns, such as happiness with positive text, and negative expressions accompanying various text sentiments; events around Trump's conviction and assassination attempt reveal partisan framing differences. The work demonstrates the value and viability of multimodal sentiment analysis for political communication and offers insights into partisan framing strategies around major events, with practical implications for social science research and AI-powered analysis of visual data.

Abstract

Sentiment analysis of textual content has become a well-established solution for analyzing social media data. However, with the rise of images and videos as primary modes of expression, more information on social media is conveyed visually. Among these, facial expressions serve as one of the most direct indicators of emotional content in images. This study analyzes a dataset of Instagram posts related to the 2024 U.S. presidential election, spanning April 5, 2024, to August 9, 2024, to compare the relationship between textual and facial sentiment. Our findings reveal that facial expressions align with text sentiment, where positive sentiment aligns with happiness, although neutral and negative facial expressions provide critical information beyond negative valence. Furthermore, during politically significant events such as Donald Trump's conviction and assassination attempt, posts depicting Trump showed a 12% increase in negative sentiment. Crucially, Democrats use their opponent's fear to depict weakness, whereas Republicans use their candidate's anger to depict resilience. Our research highlights the potential of integrating facial expression analysis with textual sentiment analysis to uncover deeper insights into social media dynamics.

Faces Speak Louder Than Words: Emotions Versus Textual Sentiment in the 2024 USA Presidential Election

TL;DR

The paper addresses how emotional content expressed in images (facial expressions) relates to textual sentiment in social media during the 2024 U.S. presidential election. It employs a multimodal pipeline that collects Instagram posts (CrowdTangle data), applies TextBlob, VADER, and DistilBERT for text, and uses Py-Feat to extract facial Action Units and classify emotions, integrating per-face analyses and a weighted cross-modal score. Key findings show that facial emotion often diverges from caption sentiment yet aligns in systemic patterns, such as happiness with positive text, and negative expressions accompanying various text sentiments; events around Trump's conviction and assassination attempt reveal partisan framing differences. The work demonstrates the value and viability of multimodal sentiment analysis for political communication and offers insights into partisan framing strategies around major events, with practical implications for social science research and AI-powered analysis of visual data.

Abstract

Sentiment analysis of textual content has become a well-established solution for analyzing social media data. However, with the rise of images and videos as primary modes of expression, more information on social media is conveyed visually. Among these, facial expressions serve as one of the most direct indicators of emotional content in images. This study analyzes a dataset of Instagram posts related to the 2024 U.S. presidential election, spanning April 5, 2024, to August 9, 2024, to compare the relationship between textual and facial sentiment. Our findings reveal that facial expressions align with text sentiment, where positive sentiment aligns with happiness, although neutral and negative facial expressions provide critical information beyond negative valence. Furthermore, during politically significant events such as Donald Trump's conviction and assassination attempt, posts depicting Trump showed a 12% increase in negative sentiment. Crucially, Democrats use their opponent's fear to depict weakness, whereas Republicans use their candidate's anger to depict resilience. Our research highlights the potential of integrating facial expression analysis with textual sentiment analysis to uncover deeper insights into social media dynamics.

Paper Structure

This paper contains 11 sections, 1 equation, 4 figures.

Table of Contents

  1. Introduction
  2. Methodology
  3. Data Acquisition
  4. Text Sentiment Analysis
  5. Image Sentiment Analysis
  6. Comparative Analysis
  7. Trump Facial Analysis
  8. Results
  9. Visual versus textual sentiment
  10. Emotional Shocks and Partisan Divergences
  11. Discussion

Figures (4)

  • Figure 1: The probability density distributions for a) Non-negative Emotions (Happy, Surprise, and Neutral) and b) Negative Emotions (Anger, Disgust, Fear, and Sadness).
  • Figure 2: Proportion results of cross-tabulation between text sentiment and facial emotion.
  • Figure 3: Comparison of Emotion Scores with 95% Confidence Intervals: DistilBERT, VADER, and TextBlob.
  • Figure 4: Changes in facial expression before and after a) President Donald Trump's conviction and b) President Trump's assassination attempt.