From Meta-Thought to Execution: Cognitively Aligned Post-Training for Generalizable and Reliable LLM Reasoning

TL;DR

The paper tackles the limitations of traditional post-training for LLM reasoning by arguing that learning complete reasoning trajectories conflates abstract problem-solving strategies with execution. It proposes a cognitively inspired two-stage pipeline: Chain-of-Meta-Thought (CoMT) to acquire meta-knowledge via abstract reasoning trajectories, and Confidence-Calibrated Reinforcement Learning (CCRL) to adapt these strategies with confidence-aware intermediate rewards, optimized via PPO. Empirical results across multiple models and eight benchmarks show consistent improvements in both in-distribution ($$2.19\%$$) and out-of-distribution ($$4.63\%$$) generalization, while reducing training time by $$65\%-70\%$$ and token usage by about $$50\%$$. The work demonstrates that aligning post-training with human cognitive principles yields not only superior generalization but also substantial efficiency gains, suggesting a practical pathway to more reliable and scalable LLM reasoning.

Abstract

Current LLM post-training methods optimize complete reasoning trajectories through Supervised Fine-Tuning (SFT) followed by outcome-based Reinforcement Learning (RL). While effective, a closer examination reveals a fundamental gap: this approach does not align with how humans actually solve problems. Human cognition naturally decomposes problem-solving into two distinct stages: first acquiring abstract strategies (i.e., meta-knowledge) that generalize across problems, then adapting them to specific instances. In contrast, by treating complete trajectories as basic units, current methods are inherently problem-centric, entangling abstract strategies with problem-specific execution. To address this misalignment, we propose a cognitively-inspired framework that explicitly mirrors the two-stage human cognitive process. Specifically, Chain-of-Meta-Thought (CoMT) focuses supervised learning on abstract reasoning patterns without specific executions, enabling acquisition of generalizable strategies. Confidence-Calibrated Reinforcement Learning (CCRL) then optimizes task adaptation via confidence-aware rewards on intermediate steps, preventing overconfident errors from cascading and improving execution reliability. Experiments across four models and eight benchmarks show 2.19\% and 4.63\% improvements in-distribution and out-of-distribution respectively over standard methods, while reducing training time by 65-70% and token consumption by 50%, demonstrating that aligning post-training with human cognitive principles yields not only superior generalization but also enhanced training efficiency.

Figures (5)

• Figure 1: Overview of our two-stage post-training framework with a concrete example from GSM8K. Stage 1 (Meta-Knowledge Acquisition): A teacher LLM generates abstract meta-thoughts excluding numerical calculations, which are used for CoMT supervised fine-tuning. Stage 2 (Task Adaptation): The CoMT-tuned model undergoes RL with rewards combining answer correctness and intermediate confidence scores (highlighted in the example: confidence on $9$ and correctness on $18$).
• Figure 2: Ablation study comparing standard outcome-based Reinforcement Learning and our Confidence-Calibrated Reinforcement Learning (CCRL). Both methods are applied to CoMT-tuned models with. CCRL consistently outperforms standard RL by incorporating confidence-aware rewards on intermediate computations, with improvements ranging from +0.2% to +2.5% across models and datasets.
• Figure 3: Overconfidence analysis on incorrect predictions. We measure the percentage of high-confidence errors at three confidence thresholds ($>0.5$, $>0.7$, $>0.9$). Confidence is calculated as described in Section \ref{['sec:ccrl_reward']}. CoMT+CCRL consistently reduces overconfidence across all models and thresholds, demonstrating that CCRL successfully teaches models appropriate confidence calibration.
• Figure 4: Generalization evaluation on GSM-Symbolic benchmark. GSM-Symbolic tests genuine reasoning capabilities through symbolic variations of mathematical problems. Models are trained on GSM8K and SVAMP, then evaluated on GSM-Symbolic. CoMT+CCRL consistently outperforms CoT-SFT+RL consistently across all models, suggesting that our framework enables more robust reasoning.
• Figure 5: Performance comparison across mathematical reasoning benchmarks. Our method (CoMT+CCRL) applied to Qwen2.5-7B achieves competitive or superior performance compared to larger base models (14B and 32B parameters) across six diverse benchmarks, demonstrating the effectiveness of our cognitively-inspired training approach in enhancing reasoning capabilities without scaling model size.