Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining
  Levels
Large Language Models (LLMs) have achieved remarkable success through
imitation learning on vast text corpora, but this paradigm creates a
training-generation gap and limits robust reasoning. Reinforcement learning
(RL) offers a more data-efficient solution capable of bridging this gap, yet
its application has been constrained by a critical data bottleneck: existing RL
datasets are orders of magnitude smaller and less diverse than web-scale
pre-training corpora. To address this, we introduce the Webscale-RL pipeline, a
scalable data engine that systematically converts large-scale pre-training
documents into millions of diverse, verifiable question-answer pairs for RL.
Using this pipeline, we construct the Webscale-RL dataset, containing 1.2
million examples across more than 9 domains. Our experiments show that the
model trained on this dataset significantly outperforms continual pretraining
and strong data refinement baselines across a suite of benchmarks. Notably, RL
training with our dataset proves substantially more efficient, achieving the
performance of continual pre-training with up to 100times fewer tokens. Our
work presents a viable path toward scaling RL to pre-training levels, enabling
more capable and efficient language models.
68e7f1de93680d7965e7c838
Zhepeng Cen
68e7f1de93680d7965e7c839
Haolin Chen
claimed_verified
2025-10-10T04:33:41.584Z
65e0c21ebd0c59bbf271004c
/avatars/3210f41fd433d798fd7857c943288625.svg
Haolin Chen
hlnchen
user
hlnchen
68e7f1de93680d7965e7c83a
Shiyu Wang
68e7f1de93680d7965e7c83b
Zuxin Liu
68e7f1de93680d7965e7c83c
Zhiwei Liu
68e7f1de93680d7965e7c83d
Ding Zhao
68e7f1de93680d7965e7c83e
Silvio Savarese
68e7f1de93680d7965e7c83f
Caiming Xiong
68e7f1de93680d7965e7c840
Huan Wang
68e7f1de93680d7965e7c841
Weiran Yao
Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining
  Levels
Large Language Models (LLMs) have achieved remarkable success through
imitation learning on vast text corpora, but this paradigm creates a
training-generation gap and limits robust reasoning. Reinforcement learning
(RL) offers a more data-efficient solution capable of bridging this gap, yet
its application has been constrained by a critical data bottleneck: existing RL
datasets are orders of magnitude smaller and less diverse than web-scale
pre-training corpora. To address this, we introduce the Webscale-RL pipeline, a
scalable data engine that systematically converts large-scale pre-training
documents into millions of diverse, verifiable question-answer pairs for RL.
Using this pipeline, we construct the Webscale-RL dataset, containing 1.2
million examples across more than 9 domains. Our experiments show that the
model trained on this dataset significantly outperforms continual pretraining
and strong data refinement baselines across a suite of benchmarks. Notably, RL
training with our dataset proves substantially more efficient, achieving the
performance of continual pre-training with up to 100times fewer tokens. Our
work presents a viable path toward scaling RL to pre-training levels, enabling
more capable and efficient language models.
A scalable data engine converts large-scale pre-training documents into diverse question-answer pairs for reinforcement learning, significantly improving model performance and efficiency.
Large Language Models imitation learning reinforcement learning Webscale-RL pipeline Webscale-RL dataset continual pretraining data refinement benchmarks