Large Language Model-Informed Feature Discovery Improves Prediction and Interpretation of Credibility Perceptions of Visual Content

TL;DR

Predicting perceived credibility of visual content and identifying driving features is addressed by an LLM-informed feature discovery workflow that uses GPT-4o to reason about visuals and captions, extract interpretable credibility-related features via targeted prompts, and incorporate these features into predictive models. The approach achieves a $r=0.76$ and $R^2=0.58$ with $MSE=0.28$, representing a $13\%$ relative improvement in $R^2$ over zero-shot predictions, and uncovers key cues such as information concreteness, caption-image alignment, and image format. SHAP analysis reveals GPT-rated credibility as a strong predictor but also highlights post- and image-level features that drive human judgments, providing interpretable insights into what makes visual content appear credible. Overall, the framework demonstrates a scalable, interpretable use of multimodal LLMs for social science, with practical implications for misinformation mitigation and visual credibility assessment.

Abstract

In today's visually dominated social media landscape, predicting the perceived credibility of visual content and understanding what drives human judgment are crucial for countering misinformation. However, these tasks are challenging due to the diversity and richness of visual features. We introduce a Large Language Model (LLM)-informed feature discovery framework that leverages multimodal LLMs, such as GPT-4o, to evaluate content credibility and explain its reasoning. We extract and quantify interpretable features using targeted prompts and integrate them into machine learning models to improve credibility predictions. We tested this approach on 4,191 visual social media posts across eight topics in science, health, and politics, using credibility ratings from 5,355 crowdsourced workers. Our method outperformed zero-shot GPT-based predictions by 13 percent in R2, and revealed key features like information concreteness and image format. We discuss the implications for misinformation mitigation, visual credibility, and the role of LLMs in social science.

Figures (5)

• Figure 1: Workflow of the study. In our proposed LLM-informed feature discovery framework, we leveraged GPT-4o’s reasoning capabilities to identify potential features underlying credibility perceptions, designed prompts to explicitly measure these features, and incorporated them to predict human credibility judgments. This approach improves both the accuracy and interpretability of credibility predictions.
• Figure 2: (A) Scatterplots illustrating the correlation between zero-shot GPT-4o–predicted credibility scores and human-perceived credibility scores (averaged at the post level). Results are presented both across all topics and separately for each individual topic. Scores range from 1 to 7, with higher values indicating greater perceived credibility and 4 representing a neutral midpoint. (B) Density plot showing the distribution of credibility scores from both GPT-4o predictions and human ratings.
• Figure 3: Overview of GPT-measured features used for analyzing social media posts, categorized into caption-level, image-level, and post-level features. The central image illustrates an example of a survey stimulus used for crowdsourcing ratings, where participants viewed a caption alongside the corresponding image below.
• Figure 4: (A) Comparison of machine learning models in predicting credibility perceptions for all issues combined. MSE = Mean Squared Error. All models were evaluated on a held-out test set (N = 400) not used during training. The bottom two models additionally included binary indicators for the eight issues as features. (B) Issue-specific comparison of a Random Forest model using GPT-measured features versus GPT-4o’s zero-shot credibility ratings. The linear model was trained separately within each issue, and evaluation was performed on that issue’s test set (N = 50).
• Figure 5: (A) SHAP values for the top 20 features in a Random Forest model using GPT-measured features to predict human-perceived credibility. (B) SHAP values for the top 20 features in a Random Forest model excluding GPT-4o’s zero-shot post credibility ratings. Features are grouped by type: post-level (P), caption-level (C), image-level (I), and topic indicators (T). Wider dot distributions indicate higher feature impact on the model's output. The color gradient represents feature values—red points on the right contribute positively to credibility perception, while blue points indicate a negative effect.