Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are UFOs Driving Innovation? The Illusion of Causality in Large Language Models

María Victoria Carro, Francisca Gauna Selasco, Denise Alejandra Mester, Mario Alejandro Leiva

TL;DR

It is found that Claude-3.5-Sonnet is the model that presents the lowest degree of causal illusion aligned with experiments on Correlation-to-Causation Exaggeration in human-written press releases, and GPT-4o-Mini remains the most robust against this cognitive bias.

Abstract

Illusions of causality occur when people develop the belief that there is a causal connection between two variables with no supporting evidence. This cognitive bias has been proposed to underlie many societal problems including social prejudice, stereotype formation, misinformation and superstitious thinking. In this research we investigate whether large language models develop the illusion of causality in real-world settings. We evaluated and compared news headlines generated by GPT-4o-Mini, Claude-3.5-Sonnet, and Gemini-1.5-Pro to determine whether the models incorrectly framed correlations as causal relationships. In order to also measure sycophantic behavior, which occurs when a model aligns with a user's beliefs in order to look favorable even if it is not objectively correct, we additionally incorporated the bias into the prompts, observing if this manipulation increases the likelihood of the models exhibiting the illusion of causality. We found that Claude-3.5-Sonnet is the model that presents the lowest degree of causal illusion aligned with experiments on Correlation-to-Causation Exaggeration in human-written press releases. On the other hand, our findings suggest that while mimicry sycophancy increases the likelihood of causal illusions in these models, especially in GPT-4o-Mini, Claude-3.5-Sonnet remains the most robust against this cognitive bias.

Are UFOs Driving Innovation? The Illusion of Causality in Large Language Models

TL;DR

It is found that Claude-3.5-Sonnet is the model that presents the lowest degree of causal illusion aligned with experiments on Correlation-to-Causation Exaggeration in human-written press releases, and GPT-4o-Mini remains the most robust against this cognitive bias.

Abstract

Illusions of causality occur when people develop the belief that there is a causal connection between two variables with no supporting evidence. This cognitive bias has been proposed to underlie many societal problems including social prejudice, stereotype formation, misinformation and superstitious thinking. In this research we investigate whether large language models develop the illusion of causality in real-world settings. We evaluated and compared news headlines generated by GPT-4o-Mini, Claude-3.5-Sonnet, and Gemini-1.5-Pro to determine whether the models incorrectly framed correlations as causal relationships. In order to also measure sycophantic behavior, which occurs when a model aligns with a user's beliefs in order to look favorable even if it is not objectively correct, we additionally incorporated the bias into the prompts, observing if this manipulation increases the likelihood of the models exhibiting the illusion of causality. We found that Claude-3.5-Sonnet is the model that presents the lowest degree of causal illusion aligned with experiments on Correlation-to-Causation Exaggeration in human-written press releases. On the other hand, our findings suggest that while mimicry sycophancy increases the likelihood of causal illusions in these models, especially in GPT-4o-Mini, Claude-3.5-Sonnet remains the most robust against this cognitive bias.

Paper Structure

This paper contains 11 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Understanding and Evaluating LLMs´ Cognitive Biases
  4. Evaluating LLMs´ Causal Capabilities
  5. Methodology
  6. Dataset Construction
  7. Tasks Configuration
  8. Evaluation Criteria
  9. Experiments and Results
  10. Limitations and Future Work
  11. Conclusion

Figures (3)

  • Figure 1: Instructions provided to the models for the first task (left) and the second task (right), along with their corresponding outputs.
  • Figure 2: Results of the first task. This figure illustrates the distribution of responses from GPT-4o-Mini, Gemini-1.5-Pro and Claude-3.5-Sonnet across the four categories of headlines.
  • Figure 3: Results of the second task. This figure illustrates the distribution of responses from GPT-4o-Mini, Gemini-1.5-Pro and Claude-3.5-Sonnet across the four categories of headlines.