PretrainZero: Reinforcement Active Pretraining

TL;DR

PretrainZero introduces a stand-alone reinforcement pretraining framework that leverages reinforcement active learning to identify and predict informative content directly from general pretraining data (Wikipedia). By coupling a mask-generation policy with a mask-prediction (CoT) model and optimizing them via a min-max GRPO objective, it overcomes the data-noise and information-density challenges of vanilla RLPT. In experiments across multiple base models, PretrainZero achieves substantial improvements in general and math reasoning during pretraining and yields further gains after RLVR-based post-training. The work demonstrates that active, verifiable RL signals on real-world corpora can meaningfully extend general reasoning capabilities without relying on reward models or supervised fine-tuning.

Abstract

Mimicking human behavior to actively learning from general experience and achieve artificial general intelligence has always been a human dream. Recent reinforcement learning (RL) based large-thinking models demonstrate impressive expert-level abilities, i.e., software and math, but still rely heavily on verifiable rewards in specific domains, placing a significant bottleneck to extend the performance boundary of general reasoning capabilities. In this work, we propose PretrainZero, a reinforcement active learning framework built on the pretraining corpus to extend RL from domain-specific post-training to general pretraining. PretrainZero features the following characteristics: 1) Active pretraining: inspired by the active learning ability of humans, PretrainZero learns a unified reasoning policy to actively identify reasonable and informative contents from pretraining corpus, and reason to predict these contents by RL. 2) Self-supervised learning: without any verifiable labels, pretrained reward models, or supervised fine-tuning, we directly pretrain reasoners from 3 to 30B base models on the general Wikipedia corpus using RL, significantly breaking the verification data-wall for general reasoning. 3) Verification scaling: by tackling increasingly challenging masked spans, PretrainZero substantially enhances the general reasoning abilities of pretrained base models. In reinforcement pretraining, PretrainZero improves Qwen3-4B-Base for 8.43, 5.96 and 10.60 on MMLU-Pro, SuperGPQA and math average benchmarks. In post-training, the pretrained models can also serve as reasoning foundation models for downstream RLVR tasks.

Figures (9)

• Figure 2: An overview of Reinforcement Active Pretraining. Compared with vanilla RLPT, PretrainZero actively explores and learns from the informative contexts on the pretraining corpus.
• Figure 3: MMLU-Pro performance for foundational RLPT methods. (a) Reinforcement next token prediction and reinforcement masked token prediction. (b) Reinforcement next token prediction with entropy and random token selection.
• Figure 4: Pretraining Mask Prediction and Mask Generation tasks with GRPO.
• Figure 5: Prompt for Mask Generation and Prediction.
• Figure 6: Training dynamic comparisons between PretrainZero and Random RLPT on Qwen3-4B-Base: (a) entropy of model outputs; (b) response length of overall samples; (c) the overall reward.