Belief Miner: A Methodology for Discovering Causal Beliefs and Causal Illusions from General Populations

TL;DR

Belief Miner addresses the challenge of measuring causal beliefs and illusions in the general population without costly laboratory experiments. It introduces a crowdsourcing methodology and interactive web interfaces that let participants construct small causal networks, which are then compared to expert-ground truth using credibility scores and crowd-based discrepancy analyses. Two climate-change focused studies (formative with 94 workers and final with 101) validate the approach, showing a mix of alignment and illusion patterns and yielding design implications for interventions and causal ML. The work provides a formal post hoc analytic framework for causal crowdsourcing, enabling scalable detection of causal beliefs and illusions with implications for science communication and decision making.

Abstract

Causal belief is a cognitive practice that humans apply everyday to reason about cause and effect relations between factors, phenomena, or events. Like optical illusions, humans are prone to drawing causal relations between events that are only coincidental (i.e., causal illusions). Researchers in domains such as cognitive psychology and healthcare often use logistically expensive experiments to understand causal beliefs and illusions. In this paper, we propose Belief Miner, a crowdsourcing method for evaluating people's causal beliefs and illusions. Our method uses the (dis)similarities between the causal relations collected from the crowds and experts to surface the causal beliefs and illusions. Through an iterative design process, we developed a web-based interface for collecting causal relations from a target population. We then conducted a crowdsourced experiment with 101 workers on Amazon Mechanical Turk and Prolific using this interface and analyzed the collected data with Belief Miner. We discovered a variety of causal beliefs and potential illusions, and we report the design implications for future research.

Figures (20)

• Figure 1: Overview of the initial collection interface. This example shows the steps participants followed to create the causal networks. In the center, we see the causal network created by the participant. The edge connecting the two nodes marked in blue is the newly created edge by the participant. We provide options to select the direction for the newly created causal edge on the right.
• Figure 2: Demographics of the crowd workers in the formative study. Y-axes represent counts for each category.
• Figure 3: Snippets of the ground truth networks created by the experts. Each expert was provided with the same template containing all causal attributes. Both the left and the right columns contain the same attributes. The experts colored green and red for expressing the upward and downward trend respectively and connected any two attributes of choice from the left column to the right column using an arrow ($\rightarrow$) to create a causal link.
• Figure 4: The Adjacency Matrix Heatmap Representation of the collected in the formative study. The cell values represent the total number of votes for that specific causal relation.
• Figure 5: The average network credibility scores and the crowd's evaluations/confidence on the causal networks collected in the formative study. (a) Distribution of Average Network Credibility Scores (0= incorrect link, 3= correct link). (b) Distribution of the crowd's provided confidence scores (1= not confident at all, 5= completely confident).