Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models

TL;DR

The paper tackles whether post-trained LLMs truly 'know' what they think by introducing Reflective Behavioral Self-Awareness and evaluating latent-policy awareness, cross-domain generalization, and faithfulness between internal reasoning traces and final outputs. It compares SFT, DPO, and GRPO across five bias/risk/reward tasks, using metrics such as $Acc_A$, $Acc_T$, and $RGR$, with GPT-4o and human verification guiding evaluation. The results show RL-based methods, especially GRPO, yield stronger self-awareness and broader generalization, but often exhibit a disconnect between thinking and saying, indicating faithfulness challenges in high-stakes scenarios. These findings inform the design of monitoring strategies for chain-of-thought reasoning and highlight the need to align internal traces with outputs to ensure trustworthy AI behavior.

Abstract

Recent advances in post-training techniques have endowed Large Language Models (LLMs) with enhanced capabilities for tackling complex, logic-intensive tasks through the generation of supplementary planning tokens. This development raises a fundamental question: Are these models aware of what they "learn" and "think"? To address this, we define three core competencies: (1) awareness of learned latent policies, (2) generalization of these policies across domains, and (3) alignment between internal reasoning traces and final outputs. We empirically evaluate these abilities on several tasks, each designed to require learning a distinct policy. Furthermore, we contrast the profiles of models post-trained via Supervised Fine-Tuning (SFT), Direct Policy Optimization (DPO), and Group Relative Policy Optimization (GRPO). Our findings indicate that RL-trained models not only demonstrate greater awareness of their learned behaviors and stronger generalizability to novel, structurally similar tasks than SFT models but also often exhibit weak alignment between their reasoning traces and final outputs, an effect most pronounced in GRPO-trained models.

Figures (3)

• Figure 1: The figure illustrates key categories of training inputs—such as bias prompts, safe vs. risky decision scenarios, and reward hacking tasks—used to fine-tune language models with SFT, DPO, and GRPO. Following training, models are evaluated using targeted prompts assessing Reflective Behavioral Self-Awareness (RQ2), Transfer of Latent policies (RQ3), and Faithfulness between model reasoning and answers (RQ1). Example model responses are shown for the evaluation tasks discussed in Section \ref{['sec:Tasks']}.
• Figure 2: Correlations between the model answers and thoughts on three evaluation settings: In-Distribution, Out-of-Distribution, and Self-Awareness on Base, SFT, DPO, GRPO and Deepseek across 4 tasks Biased, Risky (Risk Seeking), Safe (Risk Averse), and Reward (Reward Hacking).
• Figure 3: Results of Sampling Behavior across two evaluation settings: ID and OOD on SFT, DPO, GRPO and Deepseek.