GPT-4V(ision) as A Social Media Analysis Engine

TL;DR

This work benchmarks GPT-4V on five social multimedia tasks—sentiment analysis, hate speech detection, fake news identification, demographic inference, and ideology detection—using a mix of quantitative metrics and qualitative analyses. It highlights GPT-4V's strong joint image-text understanding, contextual and cultural awareness, and broad knowledge, while also documenting limitations in multilingual understanding, handling of emerging trends, and susceptibility to outdated knowledge leading to hallucinations. The study underscores the potential of large multimodal models to enhance social media analysis and user understanding, but also calls for new benchmarks and continual learning strategies to keep pace with rapidly evolving content. Overall, the findings provide a nuanced view of LMM capabilities in real-world social media contexts and outline concrete directions for future research.

Abstract

Recent research has offered insights into the extraordinary capabilities of Large Multimodal Models (LMMs) in various general vision and language tasks. There is growing interest in how LMMs perform in more specialized domains. Social media content, inherently multimodal, blends text, images, videos, and sometimes audio. Understanding social multimedia content remains a challenging problem for contemporary machine learning frameworks. In this paper, we explore GPT-4V(ision)'s capabilities for social multimedia analysis. We select five representative tasks, including sentiment analysis, hate speech detection, fake news identification, demographic inference, and political ideology detection, to evaluate GPT-4V. Our investigation begins with a preliminary quantitative analysis for each task using existing benchmark datasets, followed by a careful review of the results and a selection of qualitative samples that illustrate GPT-4V's potential in understanding multimodal social media content. GPT-4V demonstrates remarkable efficacy in these tasks, showcasing strengths such as joint understanding of image-text pairs, contextual and cultural awareness, and extensive commonsense knowledge. Despite the overall impressive capacity of GPT-4V in the social media domain, there remain notable challenges. GPT-4V struggles with tasks involving multilingual social multimedia comprehension and has difficulties in generalizing to the latest trends in social media. Additionally, it exhibits a tendency to generate erroneous information in the context of evolving celebrity and politician knowledge, reflecting the known hallucination problem. The insights gleaned from our findings underscore a promising future for LMMs in enhancing our comprehension of social media content and its users through the analysis of multimodal information.

Figures (39)

• Figure 1: In this study, we carefully select five typical social multimedia analysis tasks including sentiment analysis, hateful speech detection, fake news identification, demographic inference, and ideology detection. We adopt GPT-4V as a unified framework with different prompts (e.g., What sentiment does this combination convey?) to explore the GPT-4V's ability for social multimedia.
• Figure 2: Overview of the emerging properties of GPT-4V for social multimedia analysis tasks.
• Figure 3: Qualitative results on sentiment-infused caption generation and interpretation. GPT-4V is able to describe the image contents and interpret the original caption, with a particular emphasis on capturing the conveyed sentiment. The image descriptions and caption interpretations are highlighted. Refer to Section \ref{['sec:sentiment']} for detailed discussions.
• Figure 4: Qualitative results on image-text sentiment correlation. GPT-4V explicitly explains the interplay between the image and text. The image-text relation interpretations are highlighted. Refer to Section \ref{['sec:sentiment']} for detailed discussions.
• Figure 5: Qualitative results on contextual acuity in sentiment understandng. GPT-4V generates responses that demonstrate cultural awareness and are contextually relevant within the context of social media. The components that are related to contextual understanding are highlighted. Refer to Section \ref{['sec:sentiment']} for detailed discussions.