Revisiting Vision-Language Features Adaptation and Inconsistency for Social Media Popularity Prediction

TL;DR

Problem: SMP prediction using multimodal data may be hampered by image-text semantic misalignment in social posts. Approach: analyze semantic inconsistency across popularity levels, quantify textual quality with perplexity, and apply CLIP-adapter-based adaptation to align features with popularity, enriching the model with perplexity and similarity signals. Findings: semantic inconsistency grows with popularity; adapted text features and mismatch cues boost predictive performance, achieving SRC $0.729$ and MAE $1.227$ on SMPD. Significance: highlights the role of text quality and cross-modal mismatch in social media analytics and guides future SMP models toward robust multimodal representations.

Abstract

Social media popularity (SMP) prediction is a complex task involving multi-modal data integration. While pre-trained vision-language models (VLMs) like CLIP have been widely adopted for this task, their effectiveness in capturing the unique characteristics of social media content remains unexplored. This paper critically examines the applicability of CLIP-based features in SMP prediction, focusing on the overlooked phenomenon of semantic inconsistency between images and text in social media posts. Through extensive analysis, we demonstrate that this inconsistency increases with post popularity, challenging the conventional use of VLM features. We provide a comprehensive investigation of semantic inconsistency across different popularity intervals and analyze the impact of VLM feature adaptation on SMP tasks. Our experiments reveal that incorporating inconsistency measures and adapted text features significantly improves model performance, achieving an SRC of 0.729 and an MAE of 1.227. These findings not only enhance SMP prediction accuracy but also provide crucial insights for developing more targeted approaches in social media analysis.

Figures (5)

• Figure 1: The architecture commonly used for social media popularity prediction models is illustrated below. For every posts, Image and text descriptions are first encoded by the pre-trained Vision-Language Models (VLMs) like CLIP to extract features. These features are then aggregated with other post features (such as spatiotemporal features) and user-related features to serve as the model's input. Despite the effectiveness for SMP task, the vision-language features adaptation need to be explored, especially with semantic inconsistency between image and text description from social post.
• Figure 2: The number of training data used for analysis after data cleaning. The proportion of removed posts decreases as popularity increases, implying that the proportion of low-quality posts is relatively higher among less popular posts.
• Figure 3: The pipeline of calculating similarity between image and post description ('Title') from a post. We use a pre-trained VLM model to obtain image caption and calculate their similarity to post descriptions. This similarity serves to evaluate the semantic consistency between image and post descriptions.
• Figure 4: Cosine similarity between title and image caption features is shown on the left. The calculation process is illustrated in Figure \ref{['fig:similarity_pipeline.png']}. The right image shows the perplexity score of post descriptions, measured by pre-trained GPT-2.
• Figure 5: 3D t-SNE plot visualization of image-text features extracted by CLIP before and after adaptation, color means the density of post. Vision-language representations adaptation has a more significant impact on text features.