Exploring Conversational Agents as an Effective Tool for Measuring Cognitive Biases in Decision-Making

TL;DR

Cognitive biases significantly influence decision-making, and traditional bias measurement relies on labor-intensive, hand-crafted experiments. The authors propose a task-oriented conversational agent that can generate decision tasks and analyze user responses to detect biases such as framing and loss aversion, validated in a travel-domain pilot with significant effects ($p=0.001$) and modest-to-moderate effect sizes. The results suggest that multi-turn conversations improve detection reliability, highlighting the value of repeated measures over one-shot tasks. This work establishes a foundation for scalable, domain-agnostic bias measurement and points toward automated generation of bias tasks integrated with digital nudging strategies. Overall, it demonstrates that conversational agents can operationalize cognitive-bias assessment in natural language interactions, enabling broader deployment and cross-domain research.

Abstract

Heuristics and cognitive biases are an integral part of human decision-making. Automatically detecting a particular cognitive bias could enable intelligent tools to provide better decision-support. Detecting the presence of a cognitive bias currently requires a hand-crafted experiment and human interpretation. Our research aims to explore conversational agents as an effective tool to measure various cognitive biases in different domains. Our proposed conversational agent incorporates a bias measurement mechanism that is informed by the existing experimental designs and various experimental tasks identified in the literature. Our initial experiments to measure framing and loss-aversion biases indicate that the conversational agents can be effectively used to measure the biases.

Figures (1)

• Figure 1: Conversation Length Vs p-value and effect-size.