Using Reinforcement Learning to Train Large Language Models to Explain Human Decisions

TL;DR

The paper tackles interpretability in cognitive modeling by using reinforcement learning to elicit chain-of-thought reasoning in large language models for predicting and explaining human risky decisions. It compares three post-training strategies—SFT, Centaur-style SFT, and GRPO-based RL—on the Choices13k dataset, using LoRA-tuned Qwen-2.5-7B-Instruct. RL produces coherent CoT explanations and achieves predictive accuracy comparable to SFT, with CoTs adapting to data structure, although backbone model strength is crucial for reasoning quality. The findings demonstrate a viable path toward interpretable cognitive theories via LLMs, while highlighting computational costs and limitations such as mode collapse and reliance on pre-existing knowledge within the backbone model.

Abstract

A central goal of cognitive modeling is to develop models that not only predict human behavior but also provide insight into the underlying cognitive mechanisms. While neural network models trained on large-scale behavioral data often achieve strong predictive performance, they typically fall short in offering interpretable explanations of the cognitive processes they capture. In this work, we explore the potential of pretrained large language models (LLMs) to serve as dual-purpose cognitive models--capable of both accurate prediction and interpretable explanation in natural language. Specifically, we employ reinforcement learning with outcome-based rewards to guide LLMs toward generating explicit reasoning traces for explaining human risky choices. Our findings demonstrate that this approach produces high-quality explanations alongside strong quantitative predictions of human decisions.

Figures (12)

• Figure 1: Overview of three post-training strategies for building cognitive models of human risky choice using Qwen-2.5-7B-Instruct. The backbone LLM was first adapted using low-rank adaptation (LoRA) hu2022lora, followed by post-training via three strategies: supervised fine-tuning (SFT), Centaur-style SFT binz2024centaur, and reinforcement learning from outcome-based rewards liu2025understandingshao2024deepseekmath. In the illustrated example, the LLM is prompted to predict human choice behavior. SFT and Centaur-style models are trained directly on human data represented in JSON format. In contrast, the RL model generates candidate completions that include CoT reasoning and final predictions in JSON format, with each completion evaluated based on its predictions. Tokens or predictions relevant to each training method are highlighted in light blue.
• Figure 2: Learning curves on the (a) training and (b) test sets. Backbone LLM is Qwen-2.5-7B-Instruct. The horizontal axes indicate training epochs, while the vertical axes represent mean squared error (MSE) evaluated on the corresponding dataset. The three post-training strategies compared are supervised fine-tuning (SFT, red), Centaur-style SFT (green), and reinforcement learning (RL, grey). The lowest MSEs on the training set occur at epochs 5.98 (SFT), 5.74 (Centaur-style SFT), and 2.75 (RL), while the lowest MSEs on the test set occur at epochs 5.86 (SFT), 5.86 (Centaur-style SFT), and 2.60 (RL). (c) MSE on the test set at the final checkpoint of each post-training method. Error bars represent $\pm1$ standard error across risky-choice problems.
• Figure 3: Learning curves for the RL model. Backbone LLM is Qwen-2.5-7B-Instruct. (Left) Correctness reward, defined as one minus the absolute difference between model predictions and human choice proportions. (Middle) Format reward, based on the structure and position of the model’s JSON output. (Right) Completion length, measured by the number of generated tokens.
• Figure 4: Comparison of CoT reasoning and model predictions before (left panel) and after (right panel) RL training. Human data indicate that approximately 71% of participants selected Option B in this problem. Model completions are reformatted using Markdown for enhanced readability.
• Figure 5: Visualization of CoT reasoning generated by RL models. Each reasoning segment ("thought") is embedded using the all-MiniLM-L6-v2 model from SBERT reimers-2019-sentence-bert, followed by dimensionality reduction to two dimensions using t-SNE van2008visualizing. In the resulting 2D space, we identified nine clusters using the k-means algorithm. Each cluster is labeled with a summary of its centroid thought to provide an interpretable overview of common reasoning patterns.