MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms

TL;DR

MM-Soc addresses the need for a social multimodal evaluation by introducing a ten-task benchmark integrating datasets for misinformation, hate speech, memes, and social-context generation. It reveals that zero-shot MLLMs often perform near random on these tasks, while fine-tuning and larger models improve results, with LLaVA variants typically leading among open-source MLLMs. The benchmark also showcases two case studies—self-improvement and explanation-augmented fine-tuning—that illuminate paths for enhancing robustness and factuality. By releasing code and data, MM-Soc aims to guide future development of socially aware MLLMs and establish a more realistic evaluation standard.

Abstract

Social media platforms are hubs for multimodal information exchange, encompassing text, images, and videos, making it challenging for machines to comprehend the information or emotions associated with interactions in online spaces. Multimodal Large Language Models (MLLMs) have emerged as a promising solution to these challenges, yet they struggle to accurately interpret human emotions and complex content such as misinformation. This paper introduces MM-Soc, a comprehensive benchmark designed to evaluate MLLMs' understanding of multimodal social media content. MM-Soc compiles prominent multimodal datasets and incorporates a novel large-scale YouTube tagging dataset, targeting a range of tasks from misinformation detection, hate speech detection, and social context generation. Through our exhaustive evaluation on ten size-variants of four open-source MLLMs, we have identified significant performance disparities, highlighting the need for advancements in models' social understanding capabilities. Our analysis reveals that, in a zero-shot setting, various types of MLLMs generally exhibit difficulties in handling social media tasks. However, MLLMs demonstrate performance improvements post fine-tuning, suggesting potential pathways for improvement. Our code and data are available at https://github.com/claws-lab/MMSoc.git.

Figures (6)

• Figure 1: The MM-Soc benchmark includes 10 multimodal tasks, including 7 image-text classification tasks (misinformation detection, tagging, sarcasm, offensiveness, sentiment analysis, hate speech detection, and humor), 2 generative task (image description and social context description) and a text extraction task (OCR).
• Figure 2: Performances of the 4 representative models on the MM-Soc benchmark.
• Figure 3: Success Rate (left) and macro-F1 scores (right) of varying input lengths on PolitiFact. The instruction following abilities of MLLMs remains stable across varying input lengths, and exhibit improvements as model size increases.
• Figure 4: Left: Pairwise similarity between responses at adjacent rounds; right: similarity between response of each round and the ground-truth.
• Figure 5: Results of finetuned LLaVA-v1.5-7/13B. Compared to the zero-shot baseline, finetuning with explanations (FT w/ Expl.) and standard finetuning (FT) improves performance across different sets of tasks.