AI Credibility Signals Outrank Institutions and Engagement in Shaping News Perception on Social Media

TL;DR

This study investigates how AI-generated credibility signals influence perceived credibility and sharing of political news on social media, considering interactions with political identity and traditional engagement cues. Using a preregistered large-scale mixed-design experiment ($N = 1{,}000$) with four feedback conditions and 21 headlines per participant, it compares AI (ChatGPT) and institutional signals against a control. Results show that AI feedback, especially ChatGPT, reliably shifts accuracy and sharing judgments across ideological groups, often more robustly than institutional cues, while engagement metrics have limited effects outside native platforms. The work highlights the potential for AI-driven credibility interventions to reduce bias and enhance discernment, but also raises concerns about overreliance and the need for transparency, explainability, and user autonomy in design.

Abstract

AI-generated content is rapidly becoming a salient component of online information ecosystems, yet its influence on public trust and epistemic judgments remains poorly understood. We present a large-scale mixed-design experiment (N = 1,000) investigating how AI-generated credibility scores affect user perception of political news. Our results reveal that AI feedback significantly moderates partisan bias and institutional distrust, surpassing traditional engagement signals such as likes and shares. These findings demonstrate the persuasive power of generative AI and suggest a need for design strategies that balance epistemic influence with user autonomy.

Figures (8)

• Figure 1: A sociotechnical model of credibility perception. News consumers evaluate headline credibility based on the combined influence of social identity (ingroup alignment), engagement signals ( e.g., likes, shares), and credibility cues from institutional or AI-generated sources.
• Figure 2: Overview of experimental design. Participants were randomly assigned to one of four feedback conditions: Control, GroundNews, GroundNews Reversed, or ChatGPT Feedback. For each headline shown, social engagement cues (likes, comments, shares) were drawn at random from low, medium, or high signal levels. After viewing each headline, participants rated its credibility and shareability. The session concluded with a demographics questionnaire.
• Figure 3: Beeswarm plots with summary statistics for H1-3. Each point represents an user-input accuracy rating.
• Figure 4: Predicted accuracy (top) and shareability (bottom) ratings by headline alignment (left) and political affiliation (right). Points show group means with 95% confidence intervals. Ingroup effects are strongest among moderates, while partisans show a consistent preference for center-aligned content and limited ingroup bias in sharing.
• Figure 5: Mean Accuracy and Shareability ratings by feedback condition. Left: Estimated accuracy ratings with 95% confidence intervals. Right: Corresponding shareability ratings. Stars denote significant differences from Tukey’s HSD tests (* $p < .05$, ** $p < .01$, *** $p < .001$). All feedback increased perceived accuracy over the control, with GroundNews producing the strongest effect. ChatGPT generated the largest increase in shareability.