Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoning

TL;DR

This work investigates Reasoning CPT, a continual pretraining approach that augments domain texts with synthetic hidden thoughts to reconstruct the reasoning processes behind expert writing. By training Gemma2-9B on STEM and Law corpora with hidden thoughts and comparing to standard CPT, the study demonstrates cross-domain transfer of reasoning, larger gains on harder problems (up to ~8 points on Very Hard items), and adaptive depth of reasoning aligned to problem difficulty. The results indicate Reasoning CPT outperforms CPT across domains, with improved Pass@$k$ diversity on reasoning tasks and evidence that benefits exceed mere token-count effects. These findings suggest a scalable path to broad-domain reasoning for LLMs and point to promising future directions, including integration with RL/SFT and application to additional fields and synthetic data sources.

Abstract

Large Language Models (LLMs) have demonstrated significant improvements in reasoning capabilities through supervised fine-tuning and reinforcement learning. However, when training reasoning models, these approaches are primarily applicable to specific domains such as mathematics and programming, which imposes fundamental constraints on the breadth and scalability of training data. In contrast, continual pretraining (CPT) offers the advantage of not requiring task-specific signals. Nevertheless, how to effectively synthesize training data for reasoning and how such data affect a wide range of domains remain largely unexplored. This study provides a detailed evaluation of Reasoning CPT, a form of CPT that uses synthetic data to reconstruct the hidden thought processes underlying texts, based on the premise that texts are the result of the author's thinking process. Specifically, we apply Reasoning CPT to Gemma2-9B using synthetic data with hidden thoughts derived from STEM and Law corpora, and compare it to standard CPT on the MMLU benchmark. Our analysis reveals that Reasoning CPT consistently improves performance across all evaluated domains. Notably, reasoning skills acquired in one domain transfer effectively to others; the performance gap with conventional methods widens as problem difficulty increases, with gains of up to 8 points on the most challenging problems. Furthermore, models trained with hidden thoughts learn to adjust the depth of their reasoning according to problem difficulty.

Figures (18)

• Figure 1: Performance differences from the base model across MMLU problem difficulty levels
• Figure 2: Example of synthetic data (Law domain). Black text represents the original text. Green text indicates the hidden thoughts reconstructed from the original text.
• Figure 3: Example of synthetic data (STEM domain). Black text represents the original text. Green text indicates the hidden thoughts reconstructed from the original text.
• Figure 4: MMLU accuracy trends with training token count for (standard) CPT and Reasoning CPT. The horizontal axis shows cumulative tokens trained during training (in millions), and the vertical axis shows overall MMLU accuracy (%). Even when compared at the same token count, Reasoning CPT consistently outperforms CPT, suggesting that accuracy improvements are due to training with hidden thoughts rather than just increased token count.
• Figure 5: Relationship between problem difficulty, length of generated hidden thoughts, and accuracy. Reasoning CPT models dynamically adjust the length of their generated thoughts based on problem difficulty: they use fewer tokens than standard CPT on easy problems with comparable accuracy, and allocate significantly more tokens on hard problems with substantial accuracy gains—resulting in up to 8-point improvements at the "Very Hard" level.