PretrainZero: Reinforcement Active Pretraining
Mimicking human behavior to actively learning from general experience and achieve artificial general intelligence has always been a human dream. Recent reinforcement learning (RL) based large-thinking models demonstrate impressive expert-level abilities, i.e., software and math, but still rely heavily on verifiable rewards in specific domains, placing a significant bottleneck to extend the performance boundary of general reasoning capabilities. In this work, we propose PretrainZero, a reinforcement active learning framework built on the pretraining corpus to extend RL from domain-specific post-training to general pretraining. PretrainZero features the following characteristics: 1) Active pretraining: inspired by the active learning ability of humans, PretrainZero learns a unified reasoning policy to actively identify reasonable and informative contents from pretraining corpus, and reason to predict these contents by RL. 2) Self-supervised learning: without any verifiable labels, pretrained reward models, or supervised fine-tuning, we directly pretrain reasoners from 3 to 30B base models on the general Wikipedia corpus using RL, significantly breaking the verification data-wall for general reasoning. 3) Verification scaling: by tackling increasingly challenging masked spans, PretrainZero substantially enhances the general reasoning abilities of pretrained base models. In reinforcement pretraining, PretrainZero improves Qwen3-4B-Base for 8.43, 5.96 and 10.60 on MMLU-Pro, SuperGPQA and math average benchmarks. In post-training, the pretrained models can also serve as reasoning foundation models for downstream RLVR tasks.
693104f92d1e5b0a7d84d9f2
Xingrun Xing
admin_assigned
2025-12-04T10:39:37.521Z
651c4e0bc0247c08a46ab2a6
/avatars/3396a34ffb400f576371afc8a5064783.svg
xxr
xrxing
user
xrxing
693104f92d1e5b0a7d84d9f3
Zhiyuan Fan
claimed_verified
2025-12-04T13:49:38.606Z
65867b038dd42194878e08dc
/avatars/8965ac2f8e3ccb88759f6e4d84a30c2f.svg
Zhiyuan Fan
Zhiyuan-Fan
user
Zhiyuan-Fan
693104f92d1e5b0a7d84d9f4
Jie Lou
693104f92d1e5b0a7d84d9f5
Guoqi Li
693104f92d1e5b0a7d84d9f6
Jiajun Zhang
693104f92d1e5b0a7d84d9f7
Debing Zhang
admin_assigned
2025-12-04T10:39:57.928Z
64546be9548f22be59842cfe
/avatars/5582cf189b755d888b35f68a6e3e9bb8.svg
zdb
debingzhang
user
debingzhang
PretrainZero: Reinforcement Active Pretraining
Mimicking human behavior to actively learning from general experience and achieve artificial general intelligence has always been a human dream. Recent reinforcement learning (RL) based large-thinking models demonstrate impressive expert-level abilities, i.e., software and math, but still rely heavily on verifiable rewards in specific domains, placing a significant bottleneck to extend the performance boundary of general reasoning capabilities. In this work, we propose PretrainZero, a reinforcement active learning framework built on the pretraining corpus to extend RL from domain-specific post-training to general pretraining. PretrainZero features the following characteristics: 1) Active pretraining: inspired by the active learning ability of humans, PretrainZero learns a unified reasoning policy to actively identify reasonable and informative contents from pretraining corpus, and reason to predict these contents by RL. 2) Self-supervised learning: without any verifiable labels, pretrained reward models, or supervised fine-tuning, we directly pretrain reasoners from 3 to 30B base models on the general Wikipedia corpus using RL, significantly breaking the verification data-wall for general reasoning. 3) Verification scaling: by tackling increasingly challenging masked spans, PretrainZero substantially enhances the general reasoning abilities of pretrained base models. In reinforcement pretraining, PretrainZero improves Qwen3-4B-Base for 8.43, 5.96 and 10.60 on MMLU-Pro, SuperGPQA and math average benchmarks. In post-training, the pretrained models can also serve as reasoning foundation models for downstream RLVR tasks.
PretrainZero is a reinforcement active learning framework that enhances general reasoning capabilities by pretraining large models on a corpus without verifiable labels, improving performance on benchmarks compared to domain-specific training.
reinforcement learning RL active learning active pretraining self-supervised learning pretrained reward models Qwen3-4B-Base MMLU-Pro SuperGPQA math average benchmarks RLVR tasks