Capturing Human Cognitive Styles with Language: Towards an Experimental Evaluation Paradigm

TL;DR

This work proposes an experimental framework to validate language-based cognitive-style models by linking discourse to directly observed decision-making behavior, addressing the limitations of annotation-based ground truth. It combines a writing task with a constrained decision experiment (CIS and Inf) to induce and measure cognitive styles, yielding a Decisions dataset. The study shows that language features, especially discourse relations like causal explanations and dissonance/consonance, predict decision-making styles with an AUC around 0.8, demonstrating that cognitive style can be partly revealed through discourse patterns. The authors advocate a multimodal, behavior-grounded evaluation approach and discuss limitations and future directions for NLP research in cognitive science and human-computer interaction.

Abstract

While NLP models often seek to capture cognitive states via language, the validity of predicted states is determined by comparing them to annotations created without access the cognitive states of the authors. In behavioral sciences, cognitive states are instead measured via experiments. Here, we introduce an experiment-based framework for evaluating language-based cognitive style models against human behavior. We explore the phenomenon of decision making, and its relationship to the linguistic style of an individual talking about a recent decision they made. The participants then follow a classical decision-making experiment that captures their cognitive style, determined by how preferences change during a decision exercise. We find that language features, intended to capture cognitive style, can predict participants' decision style with moderate-to-high accuracy (AUC ~ 0.8), demonstrating that cognitive style can be partly captured and revealed by discourse patterns.

Figures (3)

• Figure 1: An alternate evaluation framework for validation of cognitive processes with language: The participants are first prompted to write about their experiences, eliciting their thought process. Then they are subjected to an experiment that would measure their behavior. The behavior is a ground truth measure of their cognitive style that can be tied to the expressed language.
• Figure 2: Randomly selected topics emerging from LDA on the participant writing describing a recent decision, depicting the types of content evoked.
• Figure 3: After participants wrote about recent decisions that they had made (Step 1 in Figure 1), they completed a decision-making experiment wherein they encountered a simulated a job offer setting (See §\ref{['subsec:job_offer']}). If the participant picks the job with higher salary and longer commute (marked in green), their preferences are expected to change in the direction of preferring high salary more, and less in the direction of preferring short commute times.