Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text
Reinforcement Learning with Verifiable Rewards (RLVR) has become a cornerstone for unlocking complex reasoning in Large Language Models (LLMs). Yet, scaling up RL is bottlenecked by limited existing verifiable data, where improvements increasingly saturate over prolonged training. To overcome this, we propose Golden Goose, a simple trick to synthesize unlimited RLVR tasks from unverifiable internet text by constructing a multiple-choice question-answering version of the fill-in-the-middle task. Given a source text, we prompt an LLM to identify and mask key reasoning steps, then generate a set of diverse, plausible distractors. This enables us to leverage reasoning-rich unverifiable corpora typically excluded from prior RLVR data construction (e.g., science textbooks) to synthesize GooseReason-0.7M, a large-scale RLVR dataset with over 0.7 million tasks spanning mathematics, programming, and general scientific domains. Empirically, GooseReason effectively revives models saturated on existing RLVR data, yielding robust, sustained gains under continuous RL and achieving new state-of-the-art results for 1.5B and 4B-Instruct models across 15 diverse benchmarks. Finally, we deploy Golden Goose in a real-world setting, synthesizing RLVR tasks from raw FineWeb scrapes for the cybersecurity domain, where no prior RLVR data exists. Training Qwen3-4B-Instruct on the resulting data GooseReason-Cyber sets a new state-of-the-art in cybersecurity, surpassing a 7B domain-specialized model with extensive domain-specific pre-training and post-training. This highlights the potential of automatically scaling up RLVR data by exploiting abundant, reasoning-rich, unverifiable internet text.
6980a86d6676f9332270678f
Ximing Lu
6980a86d6676f93322706790
David Acuna
6980a86d6676f93322706791
Jaehun Jung
6980a86d6676f93322706792
Jian Hu
claimed_verified
2026-02-03T10:08:46.047Z
63f6c04ac96958470d1e9043
/avatars/da46cdd9e21498e120ca91b67bfbfb5e.svg
Jian Hu
chuyi777
user
chuyi777
6980a86d6676f93322706793
Di Zhang
claimed_verified
2026-02-02T16:52:05.846Z
64bce15bafd1e46c5504ad38
https://cdn-avatars.huggingface.co/v1/production/uploads/64bce15bafd1e46c5504ad38/vkEjiu-mIagKlrXzDH75o.png
Di Zhang
di-zhang-fdu
user
di-zhang-fdu
6980a86d6676f93322706794
Shizhe Diao
6980a86d6676f93322706795
Yunheng Zou
claimed_verified
2026-02-04T12:33:23.270Z
658f4aac0f4519bfc25cda76
/avatars/931155c8b7cff3a885a5e7739cad2614.svg
Jackie Zou
JackieZouLM
user
JackieZouLM
6980a86d6676f93322706796
Shaokun Zhang
6980a86d6676f93322706797
Brandon Cui
6980a86d6676f93322706798
Mingjie Liu
6980a86d6676f93322706799
Hyunwoo Kim
claimed_verified
2026-03-30T09:11:52.418Z
627a8c1793d0b645835e65f0
https://cdn-avatars.huggingface.co/v1/production/uploads/1666754635237-627a8c1793d0b645835e65f0.jpeg
Hyunwoo Kim
heanu
user
heanu
6980a86d6676f9332270679a
Prithviraj Ammanabrolu
6980a86d6676f9332270679b
Jan Kautz
6980a86d6676f9332270679c
Yi Dong
6980a86d6676f9332270679d
Yejin Choi
Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text
Reinforcement Learning with Verifiable Rewards (RLVR) has become a cornerstone for unlocking complex reasoning in Large Language Models (LLMs). Yet, scaling up RL is bottlenecked by limited existing verifiable data, where improvements increasingly saturate over prolonged training. To overcome this, we propose Golden Goose, a simple trick to synthesize unlimited RLVR tasks from unverifiable internet text by constructing a multiple-choice question-answering version of the fill-in-the-middle task. Given a source text, we prompt an LLM to identify and mask key reasoning steps, then generate a set of diverse, plausible distractors. This enables us to leverage reasoning-rich unverifiable corpora typically excluded from prior RLVR data construction (e.g., science textbooks) to synthesize GooseReason-0.7M, a large-scale RLVR dataset with over 0.7 million tasks spanning mathematics, programming, and general scientific domains. Empirically, GooseReason effectively revives models saturated on existing RLVR data, yielding robust, sustained gains under continuous RL and achieving new state-of-the-art results for 1.5B and 4B-Instruct models across 15 diverse benchmarks. Finally, we deploy Golden Goose in a real-world setting, synthesizing RLVR tasks from raw FineWeb scrapes for the cybersecurity domain, where no prior RLVR data exists. Training Qwen3-4B-Instruct on the resulting data GooseReason-Cyber sets a new state-of-the-art in cybersecurity, surpassing a 7B domain-specialized model with extensive domain-specific pre-training and post-training. This highlights the potential of automatically scaling up RLVR data by exploiting abundant, reasoning-rich, unverifiable internet text.
Golden Goose synthesizes unlimited RLVR tasks from unverifiable internet text by creating multiple-choice question-answering versions of fill-in-the-middle tasks, enabling large-scale training and achieving state-of-the-art results in cybersecurity and other domains.
Reinforcement Learning with Verifiable Rewards Large Language Models fill-in-the-middle task multiple-choice question-answering unverifiable corpora GooseReason-0.7M continuous RL Qwen3-4B-Instruct GooseReason-Cyber