Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Debunking Effectiveness through LLM-based Personality Adaptation

Pietro Dell'Oglio, Alessandro Bondielli, Francesco Marcelloni, Lucia C. Passaro

Abstract

This study proposes a novel methodology for generating personalized fake news debunking messages by prompting Large Language Models (LLMs) with persona-based inputs aligned to the Big Five personality traits: Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness. Our approach guides LLMs to transform generic debunking content into personalized versions tailored to specific personality profiles. To assess the effectiveness of these transformations, we employ a separate LLM as an automated evaluator simulating corresponding personality traits, thereby eliminating the need for costly human evaluation panels. Our results show that personalized messages are generally seen as more persuasive than generic ones. We also find that traits like Openness tend to increase persuadability, while Neuroticism can lower it. Differences between LLM evaluators suggest that using multiple models provides a clearer picture. Overall, this work demonstrates a practical way to create more targeted debunking messages exploiting LLMs, while also raising important ethical questions about how such technology might be used.

Enhancing Debunking Effectiveness through LLM-based Personality Adaptation

Abstract

This study proposes a novel methodology for generating personalized fake news debunking messages by prompting Large Language Models (LLMs) with persona-based inputs aligned to the Big Five personality traits: Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness. Our approach guides LLMs to transform generic debunking content into personalized versions tailored to specific personality profiles. To assess the effectiveness of these transformations, we employ a separate LLM as an automated evaluator simulating corresponding personality traits, thereby eliminating the need for costly human evaluation panels. Our results show that personalized messages are generally seen as more persuasive than generic ones. We also find that traits like Openness tend to increase persuadability, while Neuroticism can lower it. Differences between LLM evaluators suggest that using multiple models provides a clearer picture. Overall, this work demonstrates a practical way to create more targeted debunking messages exploiting LLMs, while also raising important ethical questions about how such technology might be used.
Paper Structure (13 sections, 2 equations, 3 figures, 4 tables)

This paper contains 13 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. The Big Five Framework
  5. Tailored Debunking
  6. Persona-Based Evaluation
  7. Experiments
  8. Results and Discussion
  9. Quantitative Analysis
  10. Qualitative Analysis Summary
  11. Conclusions
  12. Limitations
  13. Examples of tailored verdicts for all 32 Profiles of a single claim.

Figures (3)

  • Figure 1: Overview of the proposed methodology for generating personality-aligned fake news debunking messages. The process involves prompting an LLM with persona-based inputs corresponding to Big Five personality traits to generate tailored debunking content. A separate LLM is employed as an evaluator to assess the psychological alignment and quality of the outputs.
  • Figure 2: Mean persuasive scores assigned by LLM-based judges across three conditions: Matched (verdict tailored to the judge's own psychological profile), Mismatched (verdicts tailored to different profiles), and Generic (non-personalized). Each persona is defined by the high presence (1) or low presence (0) of each one of the Big Five traits, in order Extraversion (E), Agreebleness (A), Conscientiousness (C), Neuroticism (N), and Openness to Experience (O).
  • Figure 3: Aggregate mean persuasiveness scores for profiles grouped by their number of positive descriptors activated